Head mounted display, method of controlling head mounted display, and computer program

ABSTRACT

A head mounted display, includes an image display unit enabling the user to visually recognize the virtual image, and an augmented reality processing unit causing the image display unit to form the virtual image including a virtual object displayed additionally to a real object actually existing in the real world, in which the augmented reality processing unit causes the virtual image including the virtual object in a first display aspect to be formed, and then causes the virtual image including the virtual object in a second display aspect to be formed after a predetermined retention time period has elapsed, and in which a degree of the visibility hindrance of the virtual object in the second display aspect for the real object is lower than a degree of the visibility hindrance of the virtual object in the first display aspect for the real object.

BACKGROUND

1. Technical Field

The present invention relates to a head mounted display.

2. Related Art

There is a technique called augmented reality in which information ispresented additionally to a real object which actually exists in thereal world, by using a computer. In the augmented reality, informationwhich is displayed additionally to a real object is also referred to asa “virtual object”. An augmented reality function is mounted in a headmounted display (hereinafter, also referred to as an “HMD”).

The HMD captures an image of external scenery with a camera, andperforms image recognition on an image through the capturing so as togenerate or acquire a virtual object. In a non-transmissive HMD in whicha visual field of a user is blocked in a state in which the user wearsthe HMD, the user visually recognizes a captured image and a virtualobject which are superimposed on each other. In a transmissive HMD inwhich the visual field of the user is not blocked in a state in whichthe user wears the HMD, only the virtual object is visually recognizedby the user. The user wearing the transmissive HMD views both a realobject of the real world and the virtual object and can thusrealistically experience the augmented reality. JP-A-2010-67083discloses a technique for realizing the augmented reality in atransmissive HMD.

The above-described virtual object is frequently disposed so as to besuperimposed on a real object or disposed around the real object. Forthis reason, there is a problem in that display of the virtual object inthe non-transmissive/transmissive HMD may hinder a user from visuallyrecognizing the real object. Such a problem is not taken intoconsideration in the techniques disclosed in JP-A-2010-67083 andJP-A-2005-38008. In addition, there is a problem in that, even in a casewhere it is considered that a virtual object is not required to bedisplayed, the virtual object is displayed, and this may hinder a userfrom visually recognizing the real object and may thus experienceinconvenience.

For this reason, a head mounted display is desirable in which display ofa virtual object does not hinder visual recognition of a real object.

SUMMARY

An advantage of some aspects of the invention is to solve at least apart of the problems described above, and the invention can beimplemented as the following forms.

(1) An aspect of the invention provides a head mounted display whichallows a user to visually recognize a virtual image. The head mounteddisplay includes an image display unit that enables the user to visuallyrecognize the virtual image; and an augmented reality processing unitthat causes the image display unit to form the virtual image including avirtual object which is displayed additionally to a real object actuallyexisting in the real world, in which the augmented reality processingunit causes the virtual image including the virtual object in a firstdisplay aspect to be formed, and then causes the virtual image includingthe virtual object in a second display aspect to be formed after apredetermined retention time period has elapsed, and in which a degreeof the visibility hindrance of the virtual object in the second displayaspect for the real object is lower than a degree of the visibilityhindrance of the virtual object in the first display aspect for the realobject.

According to the head mounted display of this aspect, the augmentedreality processing unit causes the image display unit to form thevirtual image including the virtual object in the first display aspectand then causes the image display unit to form the virtual imageincluding the virtual object in the second display aspect, having adegree of the visibility hindrance lower than a degree of the visibilityhindrance in the first display aspect after the retention time periodhas elapsed. In the above-described way, since a degree of thevisibility hindrance of the virtual object occupying the displayedvirtual image is automatically reduced after the retention time periodhas elapsed, it becomes easier for a user to visually recognize a realobject which actually exists in the real world. As a result, it ispossible to provide a head mounted display in which display of a virtualobject is unlikely to hinder visual recognition of a real object or thebackground thereof.

(2) In the head mounted display of the aspect described above, theretention time period may have a variable length.

According to the head mounted display of this aspect, depending onvarious conditions, it is possible to change a retention time period forswitching a display aspect from the first display aspect in which adegree of the visibility hindrance is high to the second display aspectin which a degree of the visibility hindrance is low, for example.

(3) The head mounted display of the aspect described above may furtherinclude a retention time acquisition unit that acquires the retentiontime period used in the augmented reality processing unit in the past,and the augmented reality processing unit may obtain a statistic of theacquired past retention time period, and may change the retention timeperiod used in the present process on the basis of the obtainedstatistic.

According to the head mounted display of this aspect, the augmentedreality processing unit can automatically change a retention time periodused in the present process on the basis of a statistic of a retentiontime period (a time period taken for a degree of the visibilityhindrance to be automatically reduced) used in the past process in theaugmented reality processing unit, that is, a tendency of the retentiontime period used in the past process.

(4) In the head mounted display of the aspect described above, theaugmented reality processing unit may obtain an information amount ofthe virtual object in the first display aspect, and may change theretention time period used in the present process on the basis of theobtained information amount.

According to the head mounted display of this aspect, the augmentedreality processing unit can change a retention time period (a timeperiod taken for a degree of the visibility hindrance to beautomatically reduced) used in the present process on the basis of aninformation amount of the virtual object in the first display aspect inwhich a degree of the visibility hindrance is high. In theabove-described way, for example, the augmented reality processing unitcan make a retention time period in a case where an information amountof a virtual object in the first display aspect is large, in otherwords, it is estimated that the user requires much time to understandcontent of a virtual object longer than a retention time period in acase where an information amount is small, and thus it is possible toimprove a user's convenience.

(5) In the head mounted display of the aspect described above, theaugmented reality processing unit may change a method of obtaining theinformation amount depending on the kind of virtual object in the firstdisplay aspect.

According to the head mounted display of this aspect, the augmentedreality processing unit can obtain an information amount of a virtualobject in the first display aspect in a method suitable for the kind ofvirtual object, and thus it is possible to understand an informationamount of the virtual object more accurately.

(6) The head mounted display of the aspect described above may furtherinclude a retention time acquisition unit that acquires a user's settingperformed on the retention time period, and the augmented realityprocessing unit may change the retention time period used in the presentprocess on the basis of the acquired user's setting.

According to the head mounted display of this aspect, the augmentedreality processing unit can change the retention time period (a timeperiod taken for a degree of the visibility hindrance to beautomatically reduced) used in the present process on the basis of auser's preference.

(7) The head mounted display of the aspect described above may furtherinclude a retention time acquisition unit that acquires retention timeinformation in which the retention time period used in the past in theaugmented reality processing unit, an information amount of the virtualobject in the first display aspect at that time, and identificationinformation for identifying the user are correlated with each other, andthe augmented reality processing unit may change the retention timeperiod used in the present process on the basis of the acquiredretention time information and the information amount of the virtualobject in the first display aspect.

According to the head mounted display of this aspect, the augmentedreality processing unit can obtain an information amount in which theuser can perform recognition per unit time by using the retention timeinformation. For this reason, the augmented reality processing unit canchange the retention time period (a time period taken for a degree ofthe visibility hindrance to be automatically reduced) used in thepresent process on the basis of, for example, the obtained informationamount (an information amount in which the user can perform recognitionper unit time) and an information amount of a virtual object in thefirst display aspect. In the above-described way, the augmented realityprocessing unit can make a retention time period in a case where aninformation amount in which the user can perform recognition per unittime is small, in other words, it is estimated that the user requiresmuch time to understand content of a virtual object longer than aretention time period in a case where an information amount is large. Asa result, the augmented reality processing unit can change the retentiontime period according to an individual preference of the user, and thusit is possible to improve a user's convenience.

(8) In the head mounted display of the aspect described above, thevirtual object in the second display aspect may include at least one oftext, a graphic, a pattern, a symbol, and a combination thereof,suggesting content of the virtual object in the first display aspect.

According to the head mounted display of this aspect, it is possible tosuggest content of a virtual object in the first display aspect by usinga virtual object in the second display aspect in which a degree of thevisibility hindrance is low.

(9) In the head mounted display of the aspect described above, theaugmented reality processing unit may stop transition from the firstdisplay aspect to the second display aspect in a case where a firstrequest is acquired from the user while waiting for the retention timeperiod to elapse.

According to the head mounted display of this aspect, since theaugmented reality processing unit can stop transition from the firstdisplay aspect to the second display aspect in response to the firstrequest from the user, it is possible to improve a user's convenience.

(10) In the head mounted display of the aspect described above, theaugmented reality processing unit may allow transition from the firstdisplay aspect to the second display aspect even before the retentiontime period has elapsed in a case where a second request is acquiredfrom the user while waiting for the retention time period to elapse.

According to the head mounted display of this aspect, the augmentedreality processing unit can force a display aspect of the augmentedreality process to transition from the first display aspect to thesecond display aspect even before a retention time period has elapsed inresponse to the second request from the user, and thus it is possible toimprove a user's convenience.

(11) The head mounted display of the aspect described above may furtherinclude a request acquisition unit that acquires a request realized byat least one of the hand, the foot, a sound and the head of the user,and a combination thereof, as the first request or the second request.

According to the head mounted display of this aspect, the user canrealize the first request or the second request by using at least one ofthe hand, the foot, a sound and the head of the user, and a combinationthereof.

(12) In the head mounted display of the aspect described above, theaugmented reality processing unit may change transition from the firstdisplay aspect to the second display aspect in stages.

According to the head mounted display of this aspect, the augmentedreality processing unit can change the transition from the first displayaspect to the second display aspect in stages and thus it is possible toreduce a sense of discomfort felt by the user due to the transition inthe display aspect.

(13) Another aspect of the invention provides a head mounted displaywhich allows a user to visually recognize a virtual image and externalscenery. The head mounted display includes an image display unit thatenables the user to visually recognize the virtual image; and anaugmented reality processing unit that causes the image display unit toform the virtual image including a virtual object which is displayedadditionally to a real object actually existing in the real world, inwhich, in response to a continuous focusing operation on the real objectduring a predetermined reference time period, the augmented realityprocessing unit causes the virtual image including the virtual object ina first display aspect to be formed, the virtual object being related tothe real object on which at least the focusing operation is performed.

According to the head mounted display of this aspect, the augmentedreality processing unit causes the image display unit to form thevirtual image including the virtual object in the first display aspectin relation to a real object on which at least a focusing operation isperformed when the focusing operation is continuously performed for apredetermined reference time period. In the above-described way, sincethe virtual object is displayed according to a user's intention such asa continuous focusing operation, a user can maintain a state in which areal object which actually exists in the real world is easily visuallyrecognized as long as the user does not continuously perform thefocusing operation. As a result, it is possible to provide a headmounted display in which display of a virtual object is unlikely tohinder visual recognition of a real object or the background thereof.

(14) In the head mounted display of the aspect described above, in acase where the virtual image including the virtual object in a seconddisplay aspect is formed prior to formation of the virtual imageincluding the virtual object in the first display aspect, in response toa continuous focusing operation on either in the virtual object in thesecond display aspect or the real object during the reference timeperiod, the augmented reality processing unit may cause the virtualimage including the virtual object in the first display aspect to beformed, the virtual object being related to the virtual object or thereal object on which at least the focusing operation is performed. Inaddition, a degree of the visibility hindrance of the virtual object inthe second display aspect for the real object may be lower than a degreeof the visibility hindrance of the virtual object in the first displayaspect for the real object.

According to the head mounted display of this aspect, after the imagedisplay unit forms the virtual image including the virtual object in thesecond display aspect, the augmented reality processing unit can causethe image display unit to form the virtual image including the virtualobject in the first display aspect, having a larger degree of visibilityhindrance than that in the second display aspect when the focusingoperation is continuously performed on a virtual object in the seconddisplay aspect for a predetermined reference time period in addition tothe real object. In the above-described way, since a display aspect ofthe virtual object transitions from the second display aspect to thefirst display aspect according to a user's intention such as acontinuous focusing operation, and thus a degree of the visibilityhindrance is reduced, the user can maintain a state in which a realobject which actually exists in the real world is easily visuallyrecognized as long as the user does not continuously perform thefocusing operation. In other words, the user can control the degree ofthe visibility hindrance of a virtual object according to the user'sintention. As a result, it is possible to provide a head mounted displayin which display of a virtual object is unlikely to hinder visualrecognition of a real object or the background thereof.

(15) The head mounted display of the aspect described above may furtherinclude a visual line acquisition unit that acquires a motion of avisual line of the user as the focusing operation.

According to the head mounted display of this aspect, the user canperform a focusing operation by using a motion of the visual linewithout moving the hand or the foot. For this reason, the user caneasily perform a focusing operation even when performing work in whichit is difficult for the user to freely use his or her hand.

(16) The head mounted display of the aspect described above may furtherinclude a motion acquisition unit that acquires a motion of the hand ofthe user as the focusing operation.

According to the head mounted display of this aspect, the user caneasily perform a focusing operation by using a motion of the hand whichis a familiar and normal action.

(17) In the head mounted display of the aspect described above, thereference time period may have a variable length.

According to the head mounted display of this aspect, it is possible tochange a reference time period for switching a display aspect to thefirst display aspect, for example, depending on various conditions.

(18) The head mounted display of the aspect described above may furtherinclude a reference time acquisition unit that acquires the referencetime period used in the augmented reality processing unit in the past,and the augmented reality processing unit may obtain a statistic of theacquired past reference time periods, and may change the reference timeperiod used in the present process on the basis of the obtainedstatistic.

According to the head mounted display of this aspect, the augmentedreality processing unit can automatically change a reference time periodused in the present process on the basis of a statistic of a referencetime period used in the past process in the augmented reality processingunit, that is, a tendency of the reference time period used in the pastprocess.

(19) In the head mounted display of the aspect described above, theaugmented reality processing unit may obtain an information amount ofthe virtual object in the first display aspect, and may change thereference time period used in the present process on the basis of theobtained information amount.

According to the head mounted display of this aspect, the augmentedreality processing unit can change a reference time period used in thepresent process on the basis of an information amount of the virtualobject in the first display aspect in which a degree of the visibilityhindrance is high. In the above-described way, for example, theaugmented reality processing unit can make a reference time period in acase where an information amount of a virtual object in the firstdisplay aspect is large, in other words, display of a virtual object intransition to the first display aspect is likely to hinder visualrecognition of a real object longer than a reference time period in acase where an information amount is small, and thus it is possible toimprove a user's convenience.

(20) In the head mounted display of the aspect described above, theaugmented reality processing unit may change the method of obtaining theinformation amount depending on the kind of virtual object in the firstdisplay aspect.

According to the head mounted display of this aspect, the augmentedreality processing unit can obtain an information amount of a virtualobject in the first display aspect in a method suitable for the kind ofvirtual object, and thus it is possible to understand an informationamount of the virtual object more accurately.

(21) The head mounted display of the aspect described above may furtherinclude a reference time acquisition unit that acquires a user's settingperformed on the reference time period, and the augmented realityprocessing unit may change the reference time period used in the presentprocess on the basis of the acquired user's setting.

According to the head mounted display of this aspect, the augmentedreality processing unit can change a reference time period used in thepresent process on the basis of a user's preference.

(22) The head mounted display of the aspect described above may furtherinclude a reference time acquisition unit that acquires reference timeinformation in which the reference time period used in the past in theaugmented reality processing unit, an information amount of the virtualobject in the first display aspect at that time, and identificationinformation for identifying the user are correlated with each other, andthe augmented reality processing unit may change the reference timeperiod used in the present process on the basis of the acquiredreference time information and the information amount of the virtualobject in the first display aspect.

According to the head mounted display of the aspect described above, theaugmented reality processing unit can obtain an information amount inwhich the user can focus per unit time by using the reference timeinformation. For this reason, the augmented reality processing unit canchange a reference time period used in the present process on the basisof, for example, the obtained information amount (an information amountin which the user can focus per unit time) and an information amount ofa virtual object in the first display aspect. In the above-describedway, the augmented reality processing unit can make a reference timeperiod in a case where an information amount in which the user can focusper unit time is small, in other words, display of a virtual object intransition to the first display aspect is likely to hinder visualrecognition of a real object longer than a reference time period in acase where an information amount is large. As a result, the augmentedreality processing unit can change a reference time period according toan individual preference of the user, and thus it is possible to improvea user's convenience.

(23) In the head mounted display of the aspect described above, thevirtual object in the second display aspect may include at least one oftext, a graphic, a pattern, a symbol, and a combination thereof,suggesting content of the virtual object in the first display aspect.

According to the head mounted display of this aspect, it is possible tosuggest content of a virtual object in the first display aspect by usinga virtual object in the second display aspect in which a degree of thevisibility hindrance is low.

(24) In the head mounted display of the aspect described above, theaugmented reality processing unit may stop transition to the firstdisplay aspect in a case where a first request is acquired from the userwhile waiting for the reference time period to elapse.

According to the head mounted display of this aspect, since theaugmented reality processing unit can stop transition to the firstdisplay aspect in response to the first request from the user, it ispossible to improve a user's convenience.

(25) In the head mounted display of the aspect described above, theaugmented reality processing unit may cause the virtual image includingthe virtual object in the first display aspect to be formed even beforethe reference time period has elapsed in a case where a second requestis acquired from the user while waiting for the reference time period toelapse.

According to the head mounted display of this aspect, the augmentedreality processing unit can cause the virtual object in the firstdisplay aspect to be displayed even before the reference time period haselapsed in response to the second request from the user, and thus it ispossible to improve a user's convenience.

(26) The head mounted display of the aspect described above may furtherinclude a request acquisition unit that acquires a request realized byat least one of the hand, the foot, a sound and the head of the user,and a combination thereof, as the first request or the second request.

According to the head mounted display of this aspect, the user canrealize the first request or the second request by using at least one ofthe hand, the foot, a sound and the head of the user, and a combinationthereof.

(27) Still another aspect of the invention provides a head mounteddisplay which allows a user to visually recognize a virtual image. Thehead mounted display includes an image display unit that enables theuser to visually recognize the virtual image; and an augmented realityprocessing unit that causes the image display unit to form the virtualimage including a virtual object which is displayed additionally to areal object actually existing in the real world, in which, in a casewhere a predetermined operation is not performed within a predeterminedreference time period, the augmented reality processing unit causes thevirtual image including the virtual object in a first display aspect tobe formed, the virtual object being related to at least the real object.

According to the head mounted display of this aspect, in a case where apredetermined operation is not started by the user within apredetermined reference time period, the augmented reality processingunit causes the image display unit to form a virtual image including avirtual object in the first display aspect. In other words, in a casewhere the predetermined operation performed by the user is startedwithin the reference time period, the augmented reality processing unitdoes not display the virtual object in the first display aspect. Forthis reason, for example, in a case where the user performs apredetermined operation (for example, certain work), it is possible toreduce a possibility that a virtual object in the first display aspectmay be displayed and may shield visual fields of the user. As a result,it is possible to provide a head mounted display in which the display ofa virtual object is unlikely to hinder visual recognition of a realobject or the background thereof.

All of the plurality of constituent elements in the respective aspectsof the invention described above are not essential, and some of theplurality of constituent elements may be changed, deleted, exchangedwith other new constituent elements, and partially deleted from thelimited content thereof, as appropriate, in order to solve some or allof the above-described problems or in order to achieve some or all ofthe effects described in the present specification. In addition, inorder to solve some or all of the above-described problems or in orderto achieve some or all of the effects described in the presentspecification, some or all of the technical features included in oneaspect of the invention described above may be combined with some or allof the technical features included in another aspect of the inventiondescribed above, and as a result may be treated as an independent aspectof the invention.

For example, one aspect of the invention may be realized as a devicewhich includes either or both of the two constituent elements includingthe image display unit and the augmented reality processing unit. Inother words, this device may or may not include the image display unit.The device may or may not include the augmented reality processing unit.This device may be realized as, for example, a head mounted display, butmay be realized as devices other than the head mounted display. Some orall of the above-described technical features of each aspect of the headmounted display are applicable to the device. For example, the device asone aspect of the invention is advantageous in that display of a virtualobject is unlikely to hinder visual recognition of a real object.However, in the device, miniaturization of the device, improvement inconvenience, achievement of low cost in manufacturing of the device,saving of resources, facilitation of manufacturing, and the like aredesirable.

The invention may be realized in various aspects, and may be realized inaspects such as a head mounted display, a control method for the headmounted display, a system including the head mounted display, a computerprogram for realizing functions of the method, the display, and thesystem, and a storage medium for storing the computer program thereon.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will be described with reference to the accompanyingdrawings, wherein like numbers reference like elements.

FIG. 1 is a diagram illustrating a schematic configuration of a headmounted display according to an embodiment of the invention.

FIG. 2 is a functional block diagram illustrating a configuration of thehead mounted display.

FIGS. 3A and 3B are diagrams illustrating an example of a virtual imagewhich is visually recognized by a user.

FIG. 4 is a diagram illustrating state transition of an augmentedreality process.

FIG. 5 is a flowchart illustrating procedures of a normal displayprocess.

FIG. 6 is a diagram illustrating an example of a normal display imagewhich is visually recognized by the user.

FIG. 7 is a flowchart illustrating procedures of a simple displayprocess.

FIG. 8 is a diagram illustrating an example of non-display.

FIG. 9 is a diagram illustrating an example of end part icon display.

FIG. 10 is a diagram illustrating an example of vicinity icon display.

FIG. 11 is a diagram illustrating an example of emphasis display.

FIG. 12 is a flowchart illustrating procedures of monitoring theestablishment of a transition condition 1.

FIG. 13 is a flowchart illustrating procedures of monitoring theestablishment of a transition condition 2.

FIGS. 14A and 14B are diagrams illustrating exterior configurations ofhead mounted displays in modification examples.

DESCRIPTION OF EXEMPLARY EMBODIMENTS A. Embodiment A-1. Configuration ofHead Mounted Display

FIG. 1 is a diagram illustrating a schematic configuration of a headmounted display according to an embodiment of the invention. A headmounted display 100 of the present embodiment is a display mounted onthe head, and is hereinafter also simply referred to as an HMD 100. TheHMD 100 is an optical transmission type head mounted display whichallows a user to visually recognize a virtual image and also to directlyvisually recognize external scenery.

The HMD 100 of the present embodiment can perform an augmented realityprocess of adding information to a “real object” which actually existsin the real world by using a CPU of the HMD 100. Here, the objectindicates any person, any plant and animal, any object (includingartificial objects, natural objects, and the like), and the like. In theaugmented reality process, information which is displayed additionallyto a real object is referred to as a “virtual object”. The HMD 100 ofthe present embodiment switches a display aspect of a virtual objectwhich is additionally presented in the augmented reality process betweena “first display aspect” and a “second display aspect”, and can thusrealize the augmented reality process so that display of a virtualobject is unlikely to hinder visual recognition of a real object. Here,a virtual object in the first display aspect has a higher degree of thevisibility hindrance than a virtual object in the second display aspect.In other words, the virtual object in the second display aspect has alower degree of the visibility hindrance than the virtual object in thefirst display aspect.

The degree of the visibility hindrance indicates a “degree of hinderingthe visibility of a user” when the user visually recognizes the realworld through a virtual image including a virtual object. The degree ofthe visibility hindrance may be replaced with a degree of visibilitysuppression.

In the present embodiment, the virtual object in the first displayaspect and the virtual object in the second display aspect take someaspects listed below so that the degree of the visibility hindrancethereof can be improved or reduced.

(A) An area which is occupied by the virtual object in a virtual imageis increased or decreased. Here, the “area occupied by the virtualobject in a virtual image” indicates an area occupied by the virtualobject in a range in which the virtual image can be shown in front ofthe eyes of a user. In this case, the virtual object in the seconddisplay aspect also includes an object by which an area occupied in avirtual image is “0”.

(B) The transmittance of at least some virtual objects is increased ordecreased. In this case, a virtual object in the second display aspectincludes an aspect in which the transmittance of all virtual objects inthe first display aspect is increased, an aspect in which only a shadowof a virtual object in the first display aspect is displayed (thetransmittance of portions other than the shadow is increased), or anaspect in which only a contour of a virtual object in the first displayaspect is displayed (the transmittance of portions other than thecontour is increased).

(C) A virtual image including a virtual object is displayed for botheyes or a single eye. Here, the display for both eyes indicates thatimage light from left and right image light generation units are emittedtoward both eyes of a user, and the display for a single eye indicatesthat image light from either of the left and right image lightgeneration units is emitted toward a single eye of the user.

In the following example, a case will be exemplified and described inwhich the aspect A is employed as a method for improving or reducingdegrees of the visibility hindrance of a virtual object in the firstdisplay aspect and a virtual object in the second display aspect.Details of the augmented reality process or details of each displayaspect will be described later.

The HMD 100 includes an image display section 20 which enables a user tovisually recognize a virtual image in a state of being mounted on thehead of the user, and a control section 10 (a controller) which controlsthe image display section 20. In the following description, forconvenience, a virtual image which is displayed by the HMD 100 and isvisually recognized by the user is also referred to as a “displayedimage”. In addition, emitting image light on the basis of image data bythe HMD 100 is also referred to as “displaying an image”.

A-1-1. Configuration of Image Display Section

FIG. 2 is a block diagram illustrating a functional configuration of theHMD 100. The image display section 20 is a mounting body which ismounted on the head of the user, and has a spectacle shape in thepresent embodiment (FIG. 1). The image display section 20 includes aright holding unit 21, a right display driving unit 22, a left holdingunit 23, a left display driving unit 24, a right optical image displayunit 26, a left optical image display unit 28, cameras 61, visual linedetection units 62, and a nine-axis sensor 66. Hereinafter, adescription will be made of positional relationships between therespective units of the image display section 20 and functions thereofin a state in which the user wears the image display section 20.

As illustrated in FIG. 1, the right optical image display unit 26 andthe left optical image display unit 28 are disposed so as to berespectively located in front of the right eye and the left eye of theuser. One end of the right optical image display unit 26 and one end ofthe left optical image display unit 28 are connected to each other atthe position corresponding to the glabella of the user. As illustratedin FIG. 2, the right optical image display unit 26 includes a rightlight guide plate 261 and a dimming plate (not illustrated). The rightlight guide plate 261 is made of a light-transmitting resin material orthe like, and guides image light output from the right display drivingunit 22 to the right eye RE of the user while reflecting the light alonga predetermined light path. The dimming plate is a thin plate-shapedoptical element, and is disposed so as to cover a surface side (anopposite side to the eye side of the user) of the image display section20. The dimming plate protects the right light guide plate 261 so as toprevent the right light guide plate 261 from being damaged,contaminated, or the like. In addition, light transmittance of thedimming plate is adjusted so as to adjust an amount of external lightentering the eyes of the user, thereby controlling an extent of visuallyrecognizing a virtual image. Further, the dimming plate may be omitted.

The left optical image display unit 28 includes a left light guide plate262 and a dimming plate (not illustrated). Details thereof are the sameas those of the right optical image display unit 26. In addition, theright optical image display unit 26 and the left optical image displayunit 28 are collectively simply referred to as “optical image displayunits”. The optical image display units may employ any method as long asa virtual image is formed in front of the eyes of a user by using imagelight, and may be realized by using, for example, a diffraction gratingor a transflective film.

As illustrated in FIG. 1, the right holding unit 21 is a member which isprovided so as to extend over a position corresponding to the temporalregion of the user from the other end part ER of the right optical imagedisplay unit 26. The left holding unit 23 is a member which is providedso as to extend over a position corresponding to the temporal region ofthe user from the other end part EL of the left optical image displayunit 28. The right holding unit 21 and the left holding unit 23 hold theimage display section 20 on the head of the user in the same manner astemples of spectacles. The right holding unit 21 and the left holdingunit 23 are collectively simply referred to as “holding units”.

As illustrated in FIG. 1, the right display driving unit 22 is disposedinside the right holding unit 21 (on a side opposing the head of theuser). The left display driving unit 24 is disposed inside the leftholding unit 23. As illustrated in FIG. 2, the right display drivingunit 22 includes a reception portion (Rx) 53, a right backlight (BL)control portion 201 and a right backlight (BL) 221 which function as alight source, a right liquid crystal display (LCD) control portion 211and a right LCD 241 which function as a display element, and a rightprojection optical system 251. The right backlight control portion 201,the right LCD control portion 211, the right backlight 221, and theright LCD 241 are also collectively referred to as an “image lightgeneration unit”. The reception portion 53 functions as a receiver whichperforms serial transmission between the control section 10 and theimage display section 20. The right backlight control portion 201 drivesthe right backlight 221 on the basis of an input control signal. Theright backlight 221 is a light emitting body such as a light emittingdiode (LED) or an electroluminescence element (EL). The right LCDcontrol portion 211 drives the right LCD 241 on the basis of a clocksignal PCLK, a vertical synchronization signal VSync, a horizontalsynchronization signal HSync, and right eye image data Data1, which areinput via the reception portion 53. The right LCD 241 is a transmissiveliquid crystal panel in which a plurality of pixels are arranged in amatrix. The right projection optical system 251 is constituted by acollimator lens which converts image light emitted from the right LCD241 into parallel beams of light flux.

The left display driving unit 24 includes a reception portion (Rx) 54, aleft backlight (BL) control portion 202 and a left backlight (BL) 222which function as a light source, a left LCD control portion 212 and aleft LCD 242 which function as a display element, and a left projectionoptical system 252. Details thereof are the same as those of the rightdisplay driving unit 22. In addition, the right display driving unit 22and the left display driving unit 24 are collectively simply referred toas “display driving units”.

As illustrated in FIG. 1, the cameras 61 are stereo cameras which arerespectively disposed at positions corresponding to the upper sides ofthe outer corners of the left and right eyes of the user. The left andright cameras 61 respectively capture images of external scenery in asurface side direction of the image display section 20, that is, in avisual line direction of the user in a state in which the HMD 100 ismounted, and acquire two images corresponding to left and right externalscenery images. The cameras 61 are visible light cameras, and theexternal scenery images acquired by the cameras 61 are images showing ashape of an object on the basis of visual light radiated from theobject. Image recognition is performed on the two external sceneryimages obtained by the cameras 61, and thus the CPU 140 of the controlsection 10 can detect and acquire motions of the hands of the user. Inorder to improve accuracy of the image recognition, the CPU 140 may usea fingertip of the user, a ring worn on the user's hand, or a specifictool held by the user, or the like, as a mark for detection. In thiscase, the cameras 61 and the CPU 140 function as a “motion acquisitionunit” which detects a motion of the user's hand. The cameras 61 in thepresent embodiment are stereo cameras, but a monocular camera may beused.

As illustrated in FIG. 1, the visual line detection units 62 arerespectively disposed at positions corresponding to lower sides of thetails of the left and right eyes of the user. Each of the left and rightvisual line detection units 62 includes an infrared light emittingportion and an infrared light receiving portion (not illustrated). Theright visual line detection unit 62 receives infrared light which isemitted from the infrared light emitting portion and is reflected aftercontacting the right eye of the user. The CPU 140 of the control section10 acquires a motion of the visual line of the right eye of the user onthe basis of the intensity of the infrared light received by the rightvisual line detection unit 62. Similarly, the left visual line detectionunit 62 receives infrared light which is emitted from the infrared lightemitting portion and is reflected after contacting the left eye of theuser. The CPU 140 acquires a motion of the visual line of the left eyeof the user on the basis of the intensity of the infrared light receivedby the left visual line detection unit 62. The reflectance of theinfrared light differs in a case where the infrared light contacts theiris, in a case where the infrared light contacts the eyelid, and in acase where the infrared light contacts the white of the eye.Specifically, the reflectance of the infrared light is the lowest valuein a case where the infrared light contacts the iris, and increases inthe order of the eyelid and the white of the eye. For this reason, theCPU 140 can acquire motions of the visual lines of the user on the basisof the intensity of the infrared light received by the visual linedetection units 62. In this case, the visual line detection units 62 andthe CPU 140 function as a “visual line acquisition unit” which acquiresmotions of visual lines of the user. In addition, the visual linedetection units 62 of the present embodiment are respectively providedon the left and right sides, but may be provided on either the left orright sides.

As illustrated in FIG. 1, the nine-axis sensor 66 is disposed at aposition corresponding to the right temple of the user. The nine-axissensor 66 is a motion sensor which detects acceleration (in three axes),angular velocity (in three axes), and geomagnetism (in three axes). Thenine-axis sensor 66 is provided in the image display section 20 and canthus detect and acquire a motion of the head of the user when the imagedisplay section 20 is mounted on the head. Here, the motion of the headincludes velocity, acceleration, angular velocity, a direction, and achange in the direction of the head.

As illustrated in FIG. 1, the image display section 20 includes acoupling unit 40 which couples the image display section 20 to thecontrol section 10. The coupling unit 40 has a main body cord 48connected to the control section 10, a right cord 42 and a left cord 44into which the main body cord 48 branches, and a connection member 46provided at the branching point. The right cord 42 is coupled to theright display driving unit 22, and the left cord 44 is coupled to theleft display driving unit 24. The connection member 46 is provided witha jack for connection to an earphone plug 30. A right earphone 32 and aleft earphone 34 extend from the earphone plug 30. A connector (notillustrated) is provided at an end of the main body cord 48 on anopposite side to the connection member 46. The connector is fitted toand unfitted from a connector (not illustrated) provided in the controlsection 10, and thus connection and disconnection between the controlsection 10 and the image display section 20 are realized. The imagedisplay section 20 and the control section 10 perform transmission ofvarious signals via the coupling unit 40. The right cord 42, the leftcord 44, and the main body cord 48 may employ, for example, a metalcable or an optical fiber.

A-1-2. Configuration of Control Section

The control section 10 is a device which controls the HMD 100. Asillustrated in FIG. 1, the control section 10 includes a determinationkey 11, a lighting unit 12, a display change key 13, a track pad 14, aluminance change key 15, a direction key 16, a menu key 17, and a powerswitch 18. The determination key 11 detects a pressing operation andoutputs a signal for determining content which is operated in thecontrol section 10. The lighting unit 12 is implemented by, for example,an LED, and performs a notification of an operation state (for example,On and Off of a power supply) of the HMD 100 by using its light emittingstate. The display change key 13 detects a pressing operation andoutputs a signal for changing a moving content image display modebetween 3D and 2D. The track pad 14 detects an operation of the fingerof the user on an operation surface of the track pad 14 and outputs asignal corresponding to detected content. As the track pad 14, varioustypes such as an electrostatic type, a pressure detection type, and anoptical type may be employed. The luminance change key 15 detects apressing operation and outputs a signal for changing luminance of theimage display section 20. The direction key 16 detects a pressingoperation on a key corresponding to upper, lower, left, and rightdirections, and outputs a signal corresponding to detected content. Thepower switch 18 detects a sliding operation of the switch, and changes apower supply state of the HMD 100.

As illustrated in FIG. 2, the control section 10 includes an inputinformation acquisition unit 110, a storage unit 120, a power supply130, a wireless communication unit 132, a GPS module 134, the CPU 140,an interface 180, and the transmission units (Tx) 51 and 52, which arecoupled to each other via a bus (not illustrated).

The input information acquisition unit 110 acquires a signal based on aninput operation which is performed with the determination key 11, thedisplay change key 13, the track pad 14, the luminance change key 15,the direction key 16, the menu key 17, and the power switch 18. Theinput information acquisition unit 110 may acquire an operation input byusing various methods other than the above description. For example, anoperation input may be acquired by using a foot switch (a switchoperated by the leg of the user). If an operation input can be acquiredby using the foot switch, the input information acquisition unit 110 canacquire an operation input from a user even when performing work inwhich it is difficult for the user to freely use his or her hands.

The storage unit 120 is constituted by a ROM, a RAM, a DRAM, a harddisk, and the like. The storage unit 120 stores various computerprograms including an operating system (OS). The storage unit 120 storesin advance a display state 121, a past retention time 122, a retentiontime setting 123, a past reference time 124, a reference time setting125, and a simple display aspect 126.

The display state 121 stores information indicating whether a displayaspect of a virtual object in the present augmented reality process isthe first display aspect or the second display aspect. The display state121 may store the type of display aspect, for example, by using a flag,or by using numbers or text.

The past retention time 122 stores history of a retention time periodused in the past augmented reality process. Here, the “retention timeperiod” indicates a time period required for a display aspect of avirtual object to transition from the first display aspect to the seconddisplay aspect in the augmented reality process. The past retention time122 may store a retention time period used in the past augmented realityprocess, an information amount of a virtual object in the first displayaspect at that time, and an identifier of a user at that time incorrelation with each other.

The retention time setting 123 stores a retention time period which isset by the user. As the content of the retention time setting 123, anyinitial value may be stored during manufacturing of the HMD 100. Thecontent of the retention time setting 123 may be changed as appropriateby the user.

The past reference time 124 stores history of a reference time periodused in the past augmented reality process. Here, the “reference timeperiod” indicates a time period required for a display aspect of avirtual object to transition from the second display aspect to the firstdisplay aspect in the augmented reality process. The past reference time124 may store a reference time period used in the past augmented realityprocess, an information amount of a virtual object in the first displayaspect at that time, and an identifier of the user at that time incorrelation with each other.

The reference time setting 125 stores a reference time period which isset by the user. As the content of the reference time setting 125, anyinitial value may be stored during manufacturing of the HMD 100. Thecontent of the reference time setting 125 may be changed as appropriateby the user.

The simple display aspect 126 stores information indicating a specificdisplay aspect which is employed in the second display aspect of avirtual object. In the present embodiment, a specific display aspectemployed in the second display aspect includes (aspect 1) end part icondisplay, (aspect 2) vicinity icon display, (aspect 3) emphasis display,and (aspect 4) non-display. Each aspect will be described later indetail. The simple display aspect 126 stores information indicating anyone of the above-described aspects 1 to 4. The simple display aspect 126may store a specific display aspect, for example, by using a flag, or byusing numbers or text.

The power supply 130 supplies power to each unit of the HMD 100. Forexample, a secondary battery may be used as the power supply 130.

The wireless communication unit 132 performs wireless communication withother apparatuses in accordance with a predetermined wirelesscommunication standard. The predetermined wireless communicationstandard refers to near field wireless communication exemplified ininfrared communication or Bluetooth (registered trademark)communication, or a wireless LAN exemplified in IEEE802.11.

The GPS module 134 receives a signal from a GPS satellite so as todetect the present position of the user of the HMD 100, and generatespresent position information indicating the present position of theuser. The present position information may be realized by, for example,coordinates indicating latitude and longitude.

The CPU 140 reads and executes the computer programs stored in thestorage unit 120 so as to function as an augmented reality processingunit 142, an OS 150, an image processing unit 160, a sound processingunit 170, and a display control unit 190.

The augmented reality processing unit 142 performs the augmented realityprocess. The augmented reality processing unit 142 includes a normaldisplay processing portion 144 and a simple display processing portion146. The normal display processing portion 144 performs a normal displayprocess which will be described later so as to cause the image displaysection 20 to form a virtual image including a virtual object in thefirst display aspect. The simple display processing portion 146 performsa simple display process which will be described later so as to causethe image display section 20 to form a virtual image including a virtualobject in the second display aspect. The augmented reality processingunit 142 performs switching between the normal display process performedby the normal display processing portion 144 and the simple displayprocess performed by the simple display processing portion 146 on thebasis of a transition condition which will be described later. In otherwords, in the present embodiment, the normal display process and thesimple display process are performed as sub-routines of the augmentedreality process.

The image processing unit 160 generates signals on the basis of content(video) which is input via the interface 180 or the wirelesscommunication unit 132. For example, in a case where the content is in adigital format, the image processing unit 160 generates a clock signalPCLK and image data Data. In addition, in the case of the digitalformat, since the clock signal PCLK is output in synchronization with animage signal, generation of a vertical synchronization signal VSync anda horizontal synchronization signal HSync and A/D conversion of ananalog image signal are not necessary. The image processing unit 160transmits, to the image display section 20, the generated clock signalPCLK, vertical synchronization signal VSync and horizontalsynchronization signal HSync, and the image data Data stored in the DRAMof the storage unit 120 via the transmission units 51 and 52. The imagedata Data which is transmitted via the transmission unit 51 is alsoreferred to as “right eye image data Data1”, and the image data Datawhich is transmitted via the transmission unit 52 is also referred to as“left eye image data Data2”. The image processing unit 160 may perform,on the image data Data stored in the storage unit 120, image processessuch as a resolution conversion process, various color tone correctionprocesses such as adjustment of luminance and color saturation, and akeystone correction process.

The display control unit 190 generates control signals for control ofthe right display driving unit 22 and the left display driving unit 24.Specifically, the display control unit 190 individually controls theright and left LCD control portions 211 and 212 to turn on and offdriving of the right and left LCDs 241 and 242, and controls the rightand left backlight control portions 201 and 202 to turn on and offdriving of the right and left backlights 221 and 222, by using thecontrol signals, so as to control each of the right display driving unit22 and the left display driving unit 24 to generate and emit imagelight. The display control unit 190 transmits the control signals to theimage display section 20 via the transmission units 51 and 52.

The sound processing unit 170 acquires an audio signal included in thecontent so as to amplify the acquired audio signal, and supplies theamplified audio signal to a speaker (not illustrated) of the rightearphone 32 and a speaker (not illustrated) of the left earphone 34.

The interface 180 performs communication with external apparatuses OA inaccordance with a predetermined wired communication standard. Thepredetermined wired communication standard refers to, for example, amicro-universal serial bus (MicroUSB), a USB, High Definition MultimediaInterface (HMDI; registered trademark), a digital visual interface(DVI), a video graphic array (VGA), a composite, Recommended Standard232C (RS-232C), or a wired LAN exemplified in IEEE802.3. As the externalapparatuses OA, for example, a personal computer PC, a portabletelephone terminal, and a gaming terminal may be used.

FIGS. 3A and 3B are diagrams illustrating examples of a virtual imagewhich is visually recognized by the user. FIG. 3A exemplifies a visualfield VR of the user in a case where the augmented reality process isnot performed. In the above-described way, image light guided to botheyes of the user of the HMD 100 forms an image on the retinae of theuser, and thus the user visually recognizes a virtual image VI. In theexample of FIG. 3A, the virtual image VI is a standby screen of the OS150 of the HMD 100. The user visually recognizes external scenery SCthrough the right optical image display unit 26 and the left opticalimage display unit 28. As mentioned above, the user of the HMD 100 ofthe present embodiment can view the virtual image VI and the externalscenery SC on the background of the virtual image VI in a portion of thevisual field VR where the virtual image VI is displayed. In a portion ofthe visual field VR where the virtual image VI is not displayed, theuser can directly view the external scenery SC through the optical imagedisplay units.

FIG. 3B exemplifies the visual field VR of the user in a case where theaugmented reality process is performed. The user visually recognizes avirtual image VI including virtual objects VO1 to VO3 through executionof the augmented reality process to be described later. The virtualobject VO1 is balloon-shaped information which is displayed around amountain (real object) of the real world in the external scenery SC. Thevirtual objects VO2 and VO3 are notebook-shaped information which isdisplayed so as to be superimposed on a tree (real object) of the realworld in the external scenery SC. As mentioned above, the user viewsboth of the virtual objects VO1 to VO3 included in the virtual image VIand the real objects in the external scenery SC which is seen throughthe virtual image VI on the background of the virtual image VI, and canthus realistically experience the augmented reality.

A-2. Augmented Reality Process

The augmented reality process is a process for presenting information(virtual object) additionally to a real object which actually exists inthe real world. The augmented reality process is started when theaugmented reality processing unit 142 receives an instruction forstarting the augmented reality process from the OS 150 or anotherapplication, or the augmented reality processing unit 142 receivesinformation indicating that power supply of the HMD 100 enters aturned-on state.

A-2-1. State Transition of Augmented Reality Process

FIG. 4 is a diagram illustrating state transition of the augmentedreality process. The augmented reality process of the present embodimentcan take a normal display state ST1 and a simple display state ST2. Inthe normal display state ST1, the augmented reality processing unit 142causes the normal display processing portion 144 to perform a normaldisplay process. As a result, a virtual image including a virtual objectin the first display aspect is formed in the image display section 20.On the other hand, in the simple display state ST2, the augmentedreality processing unit 142 causes the simple display processing portion146 to perform a simple display process. As a result, a virtual imageincluding a virtual object in the second display aspect is formed in theimage display section 20.

After the augmented reality process is started, the augmented realityprocessing unit 142 monitors establishment of a transition condition 1.The transition condition 1 is a condition for causing a state of theaugmented reality process to transition from a state after the startingto the normal display state ST1, that is, a condition for displaying avirtual object in the first display aspect. The augmented realityprocessing unit 142 may use a plurality of conditions as the transitioncondition 1, and details of the transition condition 1 will be describedlater.

In the normal display state ST1, the augmented reality processing unit142 monitors establishment of a transition condition 2 and aninvalidation action. The transition condition 2 is a condition forcausing a state of the augmented reality process to transition from thenormal display state ST1 to the simple display state ST2, that is, acondition for displaying a virtual object in the second display aspect.The augmented reality processing unit 142 may use a plurality ofconditions as the transition condition 2, and details of the transitioncondition 2 will be described later.

The invalidation action is a predetermined operation performed by theuser in order to cancel the state transition of the augmented realityprocess. In the present embodiment, a “hand waving operation” isemployed as the invalidation action. The augmented reality processingunit 142 determines whether or not a motion of the user's hand acquiredby the motion detection unit (the cameras 61 and the augmented realityprocessing unit 142 of the CPU 140) matches a pattern of a hand wavingoperation stored in advance. If the motion matches the pattern, theaugmented reality processing unit 142 determines that the invalidationaction has been performed, and if the motion does not match the pattern,the augmented reality processing unit 142 determines that theinvalidation action has not been performed. In this case, the augmentedreality processing unit 142 functions as a “request acquisition unit”,and the invalidation action functions as a “first request”.

As the invalidation action, other operations performed by using at leastone of the hand, the foot, a sound and the head of the user, and acombination thereof may be employed. As the invalidation action, forexample, an operation for forming the hand in a predetermined shape, anoperation of inputting cancellation to the control section 10, and anoperation of inputting cancellation with voice using a microphone may beemployed.

In a case where the transition condition 2 is established and theinvalidation action is not detected in the normal display state ST1, theaugmented reality processing unit 142 causes a state of the augmentedreality process to transition from the normal display state ST1 to thesimple display state ST2. On the other hand, in a case where thetransition condition 2 is established and the invalidation action isdetected in the normal display state ST1, the augmented realityprocessing unit 142 maintains a state of the augmented reality processto be the normal display state ST1.

In the above-described manner, the augmented reality processing unit 142can stop the transition from the first display aspect to the seconddisplay aspect, that is, the transition from the normal display stateST1 to the simple display state ST2 in response to the first request(invalidation action) from the user, and thus it is possible to improvea user's convenience. In addition, the augmented reality processing unit142 which functions as a request acquisition unit can acquire a requestfrom the user, performed by using at least one of the hand, the foot, asound and the head of the user, and a combination thereof, as the firstrequest.

In the simple display state ST2, the augmented reality processing unit142 monitors establishment of a transition condition 3 and aninvalidation action. The transition condition 3 is a condition forcausing a state of the augmented reality process to transition from thesimple display state ST2 to the normal display state ST1, that is, acondition for displaying a virtual object in the first display aspect.The augmented reality processing unit 142 may use a plurality ofconditions as the transition condition 3, and details of the transitioncondition 3 will be described later. The invalidation action is the sameas the invalidation action in the normal display state ST1.

In a case where the transition condition 3 is established and theinvalidation action is not detected in the simple display state ST2, theaugmented reality processing unit 142 causes a state of the augmentedreality process to transition from simple display state ST2 to thenormal display state ST1. On the other hand, in a case where thetransition condition 3 is established and the invalidation action isdetected in the simple display state ST2, the augmented realityprocessing unit 142 maintains a state of the augmented reality processto be the simple display state ST2.

In the above-described manner, the augmented reality processing unit 142can stop the transition from the second display aspect to the firstdisplay aspect, that is, the transition from the simple display stateST2 to the normal display state ST1 in response to a request(invalidation action) from the user, and thus it is possible to improvea user's convenience. In addition, the augmented reality processing unit142 which functions as a request acquisition unit can acquire a requestfrom the user, performed by using at least one of the hand, the foot, asound and the head of the user, and a combination thereof, and can stopthe transition from the second display aspect to the first displayaspect in the augmented reality processing unit 142.

A-2-2. Normal Display Process

FIG. 5 is a flowchart illustrating procedures of the normal displayprocess. The normal display process is a process for causing the imagedisplay section 20 to form a virtual image showing a virtual object inthe first display aspect. The normal display process is started on thebasis of an instruction from the augmented reality processing unit 142and is performed by the normal display processing portion 144 in thenormal display state ST1 (FIG. 4).

In step S100, the normal display processing portion 144 sets “0” to avariable i used in the process. In step S102, the normal displayprocessing portion 144 causes the cameras 61 to acquire an externalscenery image.

In step S104, the normal display processing portion 144 extracts afeature of a target object from the acquired external scenery image.Here, the “target object” indicates a “real object entering the visualfield of the user” or a “real object which is a focusing operationtarget” when the transition condition 1 or the transition condition 3 isestablished among a plurality of real objects included in the externalscenery image. The “focusing operation” indicates an operation in whichthe user focuses on a specific single point. The focusing operation maybe specified by using a visual line of the user acquired by the visualline acquisition unit (FIG. 2) or a motion of the user's hand acquiredby the motion detection unit (FIG. 2). A method of acquiring thefocusing operation will be described later.

Specifically, in step S104, the normal display processing portion 144extracts a feature of the target object included in the acquiredexternal scenery image by using an image recognition method such as a1and a2 exemplified below. The method a1 and the method a2 may becombined with each other.

(a1) An edge (feature part) of the target object is detected.

(a2) A marker (feature part) which is added to the target object inadvance is detected. As a marker added to an object, various types ofmarkers may be used, and, for example, a tape, a seal, Magic Marker(registered trademark), a laser marker, and Magic Tape (registeredtrademark) may be used. The number of markers added to an object isarbitrary.

In step S106, the augmented reality processing unit 142 acquires aposition of the target object within the visual line of the user of theHMD 100 and a distance between the HMD 100 and the target object.Specifically, the augmented reality processing unit 142 sets a positionof the feature part extracted in step S104 as a position of the targetobject within the visual line. The augmented reality processing unit 142specifies the kind of target object and a size occupied by the targetobject in the entire external scenery image on the basis of the featureextracted in step S104. The augmented reality processing unit 142estimates to what extent the target object is present at a position farfrom the HMD 100 (distance between the target object and the HMD 100) onthe basis of the specified object and size. In a case where the HMD 100includes a depth sensor or a distance measuring sensor, in step S106,the augmented reality processing unit 142 may acquire a distance betweenthe HMD 100 and the target object by using a measured value from thesensor. Therefore, the augmented reality processing unit 142 can acquirea more accurate distance.

In step S108, the augmented reality processing unit 142 acquires one ora plurality of virtual objects corresponding to the target object. Theaugmented reality processing unit 142 may acquire a virtual object froma database (not illustrated) of the HMD 100, and may acquire a virtualobject from a database (not illustrated) of another apparatus (a serveror the like) connected to the HMD 100 via a network.

In step S110, the augmented reality processing unit 142 disposes thevirtual object in accordance with the position and the distance of thetarget object so as to generate a normal display image. Specifically,the augmented reality processing unit 142 processes the virtual object(text or an image) acquired in step S108 so as to have a sizecorresponding to the distance of the target object acquired in step S106and disposes the virtual object at a position corresponding to theposition of the target object acquired in step S106. The augmentedreality processing unit 142 disposes black data in a region in which thevirtual object is not disposed in the normal display image, in order toimprove the visibility of the external scenery SC when an image isdisplayed.

FIG. 6 is a diagram illustrating an example of the normal display imagevisually recognized by the user. In step S112 of FIG. 5, the augmentedreality processing unit 142 displays the normal display image.Specifically, the augmented reality processing unit 142 transmits thenormal display image generated in step S110 to the image processing unit160. The image processing unit 160 having received the image performsthe display process described in FIG. 2. As a result, as illustrated inFIG. 6, the user can visually recognize a virtual image VI showing anormal display image NI including the virtual objects VO1 to VO3 in thevisual field VR. In addition, the user can visually recognize a targetobject (real object) decorated with the virtual objects VO1 to VO3 inexternal scenery SC on the background of the virtual image VI.

In the example illustrated in FIG. 6, the target object is a tablecutter placed in a work site. The virtual object VO1 is an image of amap indicating a component location, the virtual object VO2 is textindicating an instruction for work content given to the user, and thevirtual object VO3 is an image of an arrow assisting the instruction forthe work content. The virtual objects VO1 and VO2 are disposed aroundthe table cutter, and the virtual object VO3 is disposed so as to besuperimposed on the table cutter. As mentioned above, the augmentedreality processing unit 142 may display the virtual object around thetarget object, and may display the virtual object so as to besuperimposed on the target object. FIG. 6 illustrates an example inwhich the single target object is correlated with the three virtualobjects. However, the number of virtual objects correlated with a singletarget object is arbitrary, and may be one or more.

In step S114 of FIG. 5, in a case where the variable i is “0”, theaugmented reality processing unit 142 starts to measure a display timeperiod of the normal display image NI. In step S116, the augmentedreality processing unit 142 sets “1” to the variable i. Then, theaugmented reality processing unit 142 makes the process proceed to stepS102 and repeatedly performs the above-described processes.

As described above, in the normal display process (FIG. 5), the normaldisplay processing portion 144 can display the virtual image VIincluding the virtual objects VO1 to VO3 for giving the augmentedreality to the user of the HMD 100 on the image display section 20. Thevirtual objects VO1 to VO3 displayed in the normal display process havethe “first display aspect” in which visibility of a virtual object ofthe user is prioritized, that is, a degree of the visibility hindranceis high.

A-2-3. Simple Display Process

FIG. 7 is a flowchart illustrating procedures of the simple displayprocess. The simple display process is a process for causing the imagedisplay section 20 to form a virtual image showing a virtual object inthe second display aspect. The simple display process is started on thebasis of an instruction from the augmented reality processing unit 142and is performed by the simple display processing portion 146 in thesimple display state ST2 (FIG. 4).

In step S200, the simple display processing portion 146 finishes themeasurement of the display time period of the normal display image NI,started in step S114 of the normal display process (FIG. 5). The simpledisplay processing portion 146 stores the measured display time periodin the past retention time 122 in a form in which the time period can bedifferentiated from existing data.

In step S202, the simple display processing portion 146 acquires thesimple display aspect 126 (a specific display aspect employed in thesecond display aspect). In steps S202 and S206, the simple displayprocessing portion 146 refers to a value of the acquired simple displayaspect 126.

FIG. 8 is a diagram illustrating an example of non-display. In FIG. 7,if the simple display aspect 126 indicates “non-display” (step S202:non-display), the simple display processing portion 146 does not displaythe normal display image NI (FIG. 6) in step S204. Specifically, thesimple display processing portion 146 may not display the normal displayimage NI by using any one of b1 to b4 exemplified below.

(b1) A virtual object is suddenly erased.

The simple display processing portion 146 does not display the normaldisplay image NI in an aspect in which each of the virtual objects VO1to VO3 is suddenly erased. Specifically, the simple display processingportion 146 stops transmitting the normal display image NI to the imageprocessing unit 160. Alternatively, the simple display processingportion 146 transmits a request for stopping driving the display drivingunit (the LCD or the backlight) to the display control unit 190.Consequently, the display of the virtual image VI showing the normaldisplay image NI performed by the image display section 20 is stopped.

(b2) A virtual object is faded out to the outside of a border.

The simple display processing portion 146 does not display the normaldisplay image NI in an aspect in which each of the virtual objects VO1to VO3 is faded out to the outside of a border of the image.Specifically, the simple display processing portion 146 may repeatedlyperform generation of a normal display image in the middle of each ofthe virtual objects VO1 to VO3 being faded out to the outside of theborder of the normal display image NI and transmission of the generatedimage to the image processing unit 160. Consequently, as illustrated inFIG. 8, the position of the virtual object is gradually moved to theoutside of the border of the image (a direction indicated by the arrowin FIG. 8) every time the process is repeatedly performed. As a result,the user may observe that the virtual object seems to disappear instages to the outside of the border, and thus it is possible to reduce asense of discomfort felt by the user due to a change in the displayaspect.

(b3) A virtual object is faded out by increasing the transmittance ofthe virtual object.

The simple display processing portion 146 does not display the normaldisplay image NI in an aspect in which the transmittance of the virtualobjects VO1 to VO3 is gradually increased so that the virtual object isfaded out. Specifically, the simple display processing portion 146 mayrepeatedly perform the generation of a normal display image in which n(where n is any integer) dots are extracted from each of the virtualobjects VO1 to VO3 and transmission of the generated image to the imageprocessing unit 160. Consequently, every time the process is repeatedlyperformed, dots of the virtual object are reduced by n. As a result, theuser may observe that the virtual object seems to disappear in stagesdue to an increase in the transmission of the virtual object, and thusit is possible to reduce a sense of discomfort felt by the user due to achange in the display aspect. Instead of extracting the dots of thevirtual object, the simple display processing portion 146 may replacethe dots of the virtual object with black dots, may replace the virtualobject with a virtual object of which only a contour is displayed, mayincrease an α value of the normal display image NI, and may decreasesaturation of the normal display image NI.

(b4) A virtual object is faded out to the outside of the border whileincreasing the transmittance thereof.

This method is a combination of the method b2 and the method b3.Consequently, every time the process is repeatedly performed, the dotsof the virtual object are reduced by n, and the position of the virtualobject is also gradually moved to the outside of the border of the image(the direction indicated by the arrow in FIG. 8). As a result, the usermay observe that the virtual object seems to disappear in stages to theoutside of the border while the transmittance thereof is increased, andthus it is possible to reduce a sense of discomfort felt by the user dueto a change in the display aspect.

FIG. 9 is a diagram illustrating an example of end part icon display. InFIG. 7, if the simple display aspect 126 indicates “end part icondisplay” (step S202: others, and step S206: end part icon display), instep S210, the simple display processing portion 146 acquires one or aplurality of icon images corresponding to each of the virtual objectsVO1 to VO3 displayed on the normal display image NI (FIG. 6). The simpledisplay processing portion 146 may acquire the icon image from adatabase (not illustrated) of the HMD 100, and may acquire the iconimage from a database (not illustrated) of another apparatus (a serveror the like) connected to the HMD 100 via a network. The icon image maybe correlated with the virtual object in a one-to-one relationship, andmay be correlated with the virtual object in a one-to-many relationship,or a many-to-one relationship.

In step S212, the simple display processing portion 146 disposes allicon images acquired in step S210 at an end part so as to generate asimple display image. Here, the “end part” may be any part of the top,the bottom, the left, and the right. However, it is preferable to avoida range of about 30° in the horizontal direction and about 20° in thevertical direction, which is an effective visual used to provide a lotof information, or a range of 60° to 90° in the horizontal direction and45° to 70° in the vertical direction, which is a stable gazing field inwhich a gazing point is rapidly stabilized and viewed. The simpledisplay processing portion 146 disposes black data in a region in whichthe icon image is not disposed in the simple display image, in order toimprove the visibility of the external scenery SC when an image isdisplayed.

In step S240, the simple display processing portion 146 displays thegenerated simple display image instead of the normal display image NI(FIG. 6). As a result, as illustrated in FIG. 9, the user can visuallyrecognize a virtual image VI showing a simple display image SI includingvirtual objects (icon images) VO4 and VO5 in the visual field VR. Inaddition, the user can visually recognize a target object (real object)decorated with the virtual objects VO4 and VO5 in external scenery SC onthe background of the virtual image VI.

In the example illustrated in FIG. 9, in the same manner as in FIG. 6,the target object is a table cutter placed in a work site. The virtualobject VO4 is an icon image of a map, and the virtual object VO5 is anicon image of a manual. The virtual object VO4 is correlated with thevirtual object VO1. The virtual object VO5 is correlated with thevirtual objects VO2 and VO3. Both of the virtual objects VO4 and VO5 aredisposed at a lower right end part of the simple display image SI.

FIG. 10 is a diagram illustrating an example of vicinity icon display.In FIG. 7, if the simple display aspect 126 indicates “vicinity icondisplay” (step S202: others, and step S206: vicinity icon display), instep S220, the simple display processing portion 146 acquires one or aplurality of icon images corresponding to each of the virtual objectsVO1 to VO3 displayed on the normal display image NI (FIG. 6). Detailsthereof are the same as those in step S210.

In step S222, the simple display processing portion 146 causes thecameras 61 to acquire an external scenery image. In step S224, thesimple display processing portion 146 extracts a feature of a targetobject from the acquired external scenery image. Details thereof are thesame as those in step S104 of FIG. 5. In step S226, the simple displayprocessing portion 146 acquires a position and a distance of the targetobject. Details thereof are the same as those in step S106.

In step S228, the simple display processing portion 146 disposes theicon image in accordance with the position and the distance of thetarget object so as to generate a simple display image. Specifically,the simple display processing portion 146 processes the icon imageacquired in step S220 so as to have a size corresponding to the distanceof the target object acquired in step S226 and disposes the icon imageat a position (vicinity) corresponding to the position of the targetobject acquired in step S226. The simple display processing portion 146disposes black data in a region in which the icon image is not disposedin the simple display image, in order to improve the visibility of theexternal scenery SC when an image is displayed.

In step S240, the simple display processing portion 146 displays thegenerated simple display image instead of the normal display image NI(FIG. 6). As a result, as illustrated in FIG. 10, the user can visuallyrecognize a virtual image VI showing a simple display image SI includingvirtual objects (icon images) VO4 and VO5 in the visual field VR. In theexample illustrated in FIG. 10, the target object and the virtualobjects VO4 and VO5 are the same as those illustrated in FIG. 9. Adifference between FIGS. 10 and 9 is that the virtual objects aredisposed not at the end part of the simple display image but in thevicinity of the target object (real object).

FIG. 11 is a diagram illustrating an example of emphasis display. InFIG. 7, if the simple display aspect 126 indicates “emphasis display”(step S202: others, and step S206: emphasis display), in step S230, thesimple display processing portion 146 causes the cameras 61 to acquirean external scenery image. In step S232, the simple display processingportion 146 extracts a feature of a target object from the acquiredexternal scenery image. Details thereof are the same as those in stepS104 of FIG. 5. In step S234, the simple display processing portion 146acquires a position and a distance of the target object. Details thereofare the same as those in step S106.

In step S236, the simple display processing portion 146 generates adecorative image of the target object. Specifically, the simple displayprocessing portion 146 generates an image for decorating at least a partof the target object on the basis of the feature of the target objectspecified in step S232 and the position and the distance of the targetobject acquired in step S234. Here, the “decoration” indicates emphasis.For this reason, the “decorative image” indicates an image which causesat least a part of the target object to appear to emit light (includinglighting and blinking), an image for bordering at least a part of thetarget object, or an image which causes at least a part of the targetobject to appear to be embossed.

In step S238, the simple display processing portion 146 disposes thedecorative image acquired in step S236 in accordance with the positionand the distance of the target object so as to generate a simple displayimage. The simple display processing portion 146 disposes black data ina region in which the decorative image is not disposed in the simpledisplay image, in order to improve the visibility of the externalscenery SC when an image is displayed.

In step S240, the simple display processing portion 146 displays thegenerated simple display image instead of the normal display image NI(FIG. 6). As a result, as illustrated in FIG. 11, the user can visuallyrecognize a virtual image VI showing a simple display image SI includinga virtual object (decorative image) VO6 in the visual field VR.

In the example illustrated in FIG. 11, in the same manner as in FIG. 6,the target object (real object) is a table cutter placed in a work site.The virtual object VO6 is a decorative image for emphasizing a part ofthe table cutter, that is, a blade part.

After step S240 of FIG. 7 is finished, the simple display processingportion 146 makes the process proceed to step S202, and repeatedlyperforms the above-described processes.

As mentioned above, in the simple display process (FIG. 7), in a casewhere the simple display aspect 126 indicates anyone of the “end particon display”, the “vicinity icon display”, and the “emphasis display”,the simple display processing portion 146 can display the virtual imageVI including the virtual objects VO4 to VO6 for giving the augmentedreality to the user of the HMD 100 on the image display section 20. Thevirtual objects VO4 to VO6 displayed in the simple display process havethe “second display aspect” in which visibility of external scenery ofthe user is prioritized, that is, a degree of the visibility hindranceis low.

As described above, according to the simple display process (FIG. 7), itis possible to suggest content of the virtual objects in the firstdisplay aspect by using the virtual objects (VO4 to VO6 in FIGS. 8, 9,10 and 11) in the second display aspect, having a smaller occupied areaby the virtual object in the virtual image VI than the area occupied bythe virtual objects (VO1 to VO3 in FIG. 6) in the first display aspect.

In each description of the end part icon display, the vicinity icondisplay, and the emphasis display, transition between the normal displayimage NI (FIG. 6) and the simple display image SI (FIGS. 9, 10 and 11)is suddenly performed. However, transition from the normal display imageNI to the simple display image SI may be gradually performed.Specifically, a method such as the methods b2 to b4 of step S204 may beused. In the above-described way, the user may observe that the normaldisplay image NI appears to change to the simple display image SI instages, and thus it is possible to reduce a sense of discomfort felt bythe user due to a change in the display aspect.

In each description of the end part icon display and the vicinity icondisplay, the virtual objects VO4 and VO5 are icon images. In thedescription of the emphasis display, the virtual object VO6 is a graphicimage. However, the virtual objects in the second display aspect may notnecessarily be an icon image or a graphic image. A virtual object in thesecond display aspect may employ any form as long as text, a graphic, apattern, a symbol, and a combination thereof can suggest content of thevirtual objects (VO1 to VO3 in FIG. 6) in the first display aspect. Forexample, a virtual object in the second display aspect may be simpletext, may be a simple symbol, and may be a combination of a pattern andtext.

Hereinafter, a description will be made of procedures of monitoring theestablishment of the transition conditions 1 to 3, performed by theaugmented reality processing unit 142 in the augmented reality process.

A-2-4. Monitoring of Establishment of Transition Condition 1

FIG. 12 is a flowchart illustrating procedures of monitoring theestablishment of the transition condition 1. Among at least one ofconditions 1-1 to 1-5 listed below is set in the augmented realityprocessing unit 142 in advance as the transition condition 1. Theaugmented reality processing unit 142 determines that the transitioncondition 1 is established in a case where any one of the set conditions1-1 to 1-5 is established.

(1-1) In a case where a real object which is a display target of avirtual object enters a visual field of the user

(1-2) In a case where a focusing operation performed by the user iscontinuously performed for a time period which is equal to or greaterthan a statistic of a reference time period in the past augmentedreality process

(1-3) In a case where a focusing operation performed by the user iscontinuously performed for a time period which is equal to or longerthan a time period which is obtained on the basis of an informationamount of a virtual object

(1-4) In a case where a focusing operation performed by the user iscontinuously performed for a time period which is equal to or longerthan a value which is set by the user

(1-5) In a case where a focusing operation performed by the user iscontinuously performed for a time period which is equal to or longerthan a time period which is obtained by taking into consideration anindividual preference of the user and an information amount of a virtualobject

Next, with reference to FIG. 12, a description will be made of specificprocedures for determining establishment of the conditions 1-1 to 1-5 inthe augmented reality processing unit 142.

(1-1) CASE (within visual field): In a case where a real object which isa display target of a virtual object enters the visual field of the user

In step S310, the augmented reality processing unit 142 acquires anexternal scenery image by using the cameras 61. In step S312, theaugmented reality processing unit 142 performs image recognition on theacquired external scenery image so as to determine whether or not a realobject which is a display target of a virtual object is included in theexternal scenery image. The “real object which is a display target of avirtual object” is the “target object” in the normal display process(FIG. 5).

If the target object is included therein (step S312: YES), the augmentedreality processing unit 142 determines that the transition condition 1is established in step S314. If the target object is not includedtherein, the augmented reality processing unit 142 continues to monitorestablishment of the conditions 1-1 to 1-5.

As mentioned above, if the condition 1-1 is used, the augmented realityprocessing unit 142 can determine that the transition condition 1 isestablished in a case where a real object (target object) which is adisplay target of a virtual object enters a visual field of the user,and can cause a state (FIG. 4) of the augmented reality process totransition from an initial state to the normal display state ST1. As aresult, the normal display process (FIG. 5) is performed by the normaldisplay processing portion 144, and the HMD 100 allows the user tovisually recognize a virtual image VI (FIG. 6) including a virtualobject in the first display aspect.

(1-2) CASE (past reference time): In a case where a focusing operationperformed by the user is continuously performed for a time period whichis equal to or longer than a statistic of a reference time period in thepast augmented reality process

In step S320, the augmented reality processing unit 142 detects startingof a focusing operation (an operation in which the user focuses on aspecific point) performed by the user. In the present embodiment, in acase where a visual line of the user acquired by the visual lineacquisition unit (the visual line detection unit 62 and the augmentedreality processing unit 142 of the CPU 140) is not moved from a certainpoint for a predetermined time period or more, it may be determined thata focusing operation has been started. The predetermined time period maybe set to any value. In the determination of whether or not a visualline is “moved from a certain point”, it is preferable to allow shiftswithin a predetermined range in consideration of shifts of a visual linedue to nystagmus. In a case where the hand of the user acquired by themotion detection unit (the cameras 61 and the augmented realityprocessing unit 142 of the CPU 140) is not moved from a certain pointfor a predetermined time period or more, it may be determined that afocusing operation has been started. Also in this case, in thedetermination of whether or not the hand is “moved from a certainpoint”, it is preferable to allow shifts within a predetermined range inconsideration of hand shaking.

In step S322, the augmented reality processing unit 142 acquires notonly history of a reference time period (hereinafter, also referred toas a “past reference time period”) used in the past augmented realityprocess, stored in the past reference time 124 but also historyassociated with other users. In step S322, the augmented realityprocessing unit 142 functions as a “reference time acquisition unit”,and the past reference time 124 functions as “reference timeinformation”.

In step S324, the augmented reality processing unit 142 obtains astatistic of the past reference time periods by using the acquiredhistory. The statistic may be obtained by using any statistical method,and may be, for example, an average value, a mode, or a median. Theaugmented reality processing unit 142 sets the obtained statistic as a“reference time period used in the present process”.

In step S326, the augmented reality processing unit 142 determineswhether or not a continuous time period of the focusing operation of theuser of which the starting is detected in step S320 is equal to orlonger than the reference time period used in the present process (thestatistic in step S324).

If the continuous time period of the focusing operation is equal to orlonger than the statistics (step S326: YES), in step S328, the augmentedreality processing unit 142 determines that the transition condition 1is established. The augmented reality processing unit 142 stores anactual continuous time period of the focusing operation of the user ofwhich starting is detected in step S320, an information amount of avirtual object in the first display aspect, and an identifier of theuser in the past reference time 124. A method of obtaining aninformation amount of a virtual object in the first display aspect willbe described in step S332.

In step S328, the augmented reality processing unit 142 specifies a realobject which is a target of the focusing operation. Specifically, theaugmented reality processing unit 142 may specify a real object which isa target of the focusing operation by checking a direction of the visualline of the user detected in step S320 with the external scenery imageobtained by the cameras 61. The “real object which is a target of thefocusing operation” is a “target object” in the normal display process(FIG. 5) and the simple display process (FIG. 7). In a case of using amotion of the hand of the user in step S320, the augmented realityprocessing unit 142 may perform image analysis on the external sceneryimage obtained by the cameras 61, and may specify an object indicated bythe hand (for example, a fingertip) of the user as a real object whichis a target of the focusing operation.

In a case where the focusing operation is finished before the statisticor greater is reached, the augmented reality processing unit 142continues to monitor establishment of the conditions 1-1 to 1-5.

As mentioned above, if the condition 1-2 is used, the augmented realityprocessing unit 142 can automatically change a reference time periodused in the present augmented reality process on the basis of astatistic of a reference time period (past reference time period) usedin the past augmented reality process, that is, a tendency of the pastreference time period. In a case where a continuous time period of afocusing operation of the user is equal to or longer than a referencetime period used in the present augmented reality process, the augmentedreality processing unit 142 determines that the transition condition 1is established and can thus cause a state (FIG. 4) of the augmentedreality process to transition from the initial state to the normaldisplay state ST1.

(1-3) CASE (information amount): In a case where a focusing operationperformed by the user is continuously performed for a time period whichis equal to or longer than a time period which is obtained on the basisof an information amount of a virtual object

In step S330, the augmented reality processing unit 142 detects startingof a focusing operation performed by the user. Details thereof are thesame as those in step S320.

In step S332, the augmented reality processing unit 142 acquires aninformation amount of a virtual object. Specifically, the augmentedreality processing unit 142 specifies a real object (that is, a targetobject) which is a target of the focusing operation detected in stepS320. Details thereof are the same as those in step S326. The augmentedreality processing unit 142 acquires one or a plurality of virtualobjects (virtual objects in the first display aspect) corresponding tothe specified target object. Details thereof are the same as those instep S108 of FIG. 5. The augmented reality processing unit 142 obtainsan amount of information regarding one or the plurality of acquiredvirtual objects. The augmented reality processing unit 142 may obtain aninformation amount of the virtual object, for example, by using any oneof methods c1 to c3 described below. In a case where a plurality ofvirtual objects are acquired, as an information amount, an average valueof amounts of information regarding the plurality of virtual objects maybe used, or a sum value of amounts of information regarding theplurality of virtual objects may be used.

(c1) A file size of a virtual object: The augmented reality processingunit 142 preferably employs the method c1 in a case where a virtualobject is formed of text and an image, in a case where a virtual objectis a video, and in a case where the kind of virtual object is unclear.

(c2) The number of letters included in a virtual object: The augmentedreality processing unit 142 preferably employs the method c2 in a casewhere a virtual object is text.

(c3) A ratio of black dots when a virtual object is binarized: Theaugmented reality processing unit 142 preferably employs the method c3in a case where a virtual object is an image.

If the above-described methods c1 to c3 are used, the augmented realityprocessing unit 142 can obtain an information amount of a virtual objectin the first display aspect in a method suitable for the kind of virtualobject, and thus it is possible to understand an information amount ofthe virtual object more accurately.

In step S334, the augmented reality processing unit 142 obtains athreshold value of a reference time period used in the present processon the basis of the information amount of the virtual object acquired instep S332. The threshold value may be obtained by using any method, and,for example, a value obtained by multiplying an information amount by apredetermined coefficient may be used as the threshold value, and thethreshold value may be obtained by using a table in which informationamount candidates are correlated with threshold value candidates. Theaugmented reality processing unit 142 sets the obtained threshold valueas a “reference time period used in the present process”.

In step S336, the augmented reality processing unit 142 determineswhether or not a continuous time period of the focusing operation of theuser of which the starting is detected in step S330 is equal to orlonger than the reference time period used in the present process (thethreshold value in step S334).

If the continuous time period of the focusing operation is equal to orgreater than the threshold value (step S336: YES), in step S338, theaugmented reality processing unit 142 determines that the transitioncondition 1 is established. Processes (storing in the past referencetime 124, and specifying of a real object which is a target of thefocusing operation) after the transition condition 1 is established arethe same as those in step S328. In a case where the focusing operationis finished before the threshold value or greater is reached, theaugmented reality processing unit 142 continues to monitor establishmentof the conditions 1-1 to 1-5.

As mentioned above, if the condition 1-3 is used, the augmented realityprocessing unit 142 can change a reference time period used in thepresent augmented reality process on the basis of an information amountof a virtual object (VO1 to VO3 in FIG. 6) in the first display aspect,which occupies a large area in a virtual image VI (a degree of thevisibility hindrance is high). In the above-described way, for example,the augmented reality processing unit 142 can make a reference timeperiod in a case where an information amount of a virtual object in thefirst display aspect is large, in other words, display of a virtualobject in transition from an initial state to the first display aspectis likely to hinder visual recognition of a real object longer than areference time period in a case where an information amount is small,and thus it is possible to improve a user's convenience. In addition, ina case where a continuous time period of a focusing operation performedby the user is equal to or longer than a reference time period used inthe present augmented reality process, the augmented reality processingunit 142 determines that the transition condition 1 is established, andcan thus cause a state (FIG. 4) of the augmented reality process totransition to the normal display state ST1.

(1-4) CASE (user's setting): In a case where a focusing operationperformed by the user is continuously performed for a time period whichis equal to or longer than a value which is set by the user

In step S340, the augmented reality processing unit 142 detects startingof a focusing operation performed by the user. Details thereof are thesame as those in step S320.

In step S342, the augmented reality processing unit 142 acquires a setvalue of a reference time period which is set by the user, stored in thereference time setting 125. The augmented reality processing unit 142functions as a “reference time acquisition unit” in step S342. Theaugmented reality processing unit 142 sets the acquired set value as a“reference time period used in the present process”.

In step S344, the augmented reality processing unit 142 determineswhether or not a continuous time period of the focusing operation of theuser of which the starting is detected in step S340 is equal to orlonger than the reference time period used in the present process (theset value in step S342).

If the continuous time period of the focusing operation is equal to orgreater than the set value (step S344: YES), in step S346, the augmentedreality processing unit 142 determines that the transition condition 1is established. Processes (storing in the past reference time 124, andspecifying of a real object which is a target of the focusing operation)after the transition condition 1 is established are the same as those instep S328. In a case where the focusing operation is finished before theset value or greater is reached, the augmented reality processing unit142 continues to monitor establishment of the conditions 1-1 to 1-5.

As mentioned above, if the condition 1-4 is used, the augmented realityprocessing unit 142 can change a reference time period used in thepresent augmented reality process on the basis of a set value which ispreferred by the user and is stored in the reference time setting 125.In addition, in a case where a continuous time period of a focusingoperation performed by the user is equal to or longer than a referencetime period used in the present augmented reality process, the augmentedreality processing unit 142 determines that the transition condition 1is established, and can thus cause a state (FIG. 4) of the augmentedreality process to transition from an initial state to the normaldisplay state ST1.

(1-5) CASE (individual preference of the user): In a case where afocusing operation performed by the user is continuously performed for atime period which is equal to or longer than a time period which isobtained by taking into consideration an individual preference of theuser and an information amount of a virtual object

In step S350, the augmented reality processing unit 142 detects startingof a focusing operation performed by the user. Details thereof are thesame as those in step S320.

In step S352, the augmented reality processing unit 142 acquires historyassociated with the present user of the HMD 100 on the basis of historyof a reference time period used in the past augmented reality process,stored in the past reference time 124. The augmented reality processingunit 142 may search the past reference time 124 with an identifier ofthe user as a key. The augmented reality processing unit 142 functionsas a “reference time acquisition unit” in step S352.

In step S354, the augmented reality processing unit 142 obtains aninformation amount in which the user of the HMD 100 can focus per unittime by dividing an “information amount” of the acquired history by the“continuous time period of the focusing operation”. Next, the augmentedreality processing unit 142 obtains an ideal reference time period bydividing an information amount of a virtual object in the first displayaspect by the obtained information amount (the information amount inwhich the user can focus per unit time). The augmented realityprocessing unit 142 sets the obtained ideal reference time period as a“reference time period used in the present process”. A method ofobtaining an information amount of a virtual object in the first displayaspect is the same as in step S332.

In step S356, the augmented reality processing unit 142 determineswhether or not a continuous time period of the focusing operation of theuser of which the starting is detected in step S350 is equal to orlonger than the reference time period used in the present process (theideal reference time period in step S354).

If the continuous time period of the focusing operation is equal to orlonger than the ideal reference time period (step S356: YES), in stepS358, the augmented reality processing unit 142 determines that thetransition condition 1 is established. Processes (storing in the pastreference time 124, and specifying of a real object which is a target ofthe focusing operation) after the transition condition 1 is establishedare the same as those in step S328. In a case where the focusingoperation is finished before the ideal reference time period or greateris reached, the augmented reality processing unit 142 continues tomonitor establishment of the conditions 1-1 to 1-5.

As mentioned above, if the condition 1-5 is used, the augmented realityprocessing unit 142 can obtain an information amount in which the usercan focus per unit time by using the reference time information (thepast reference time 124). For this reason, the augmented realityprocessing unit 142 can change a reference time period used in thepresent augmented reality process on the basis of, for example, theobtained information amount (an information amount in which the user canfocus per unit time) and an information amount of a virtual object (VO1to VO3 in FIG. 6) in the first display aspect. In the above-describedway, the augmented reality processing unit 142 can make a reference timeperiod in a case where an information amount in which the user can focusper unit time is small, in other words, display of a virtual object intransition from an initial state to the first display aspect is likelyto hinder visual recognition of a real object longer than a referencetime period in a case where an information amount is large. As a result,the augmented reality processing unit 142 can change a reference timeperiod according to an individual preference of the user, and thus it ispossible to improve a user's convenience. In addition, in a case where acontinuous time period of a focusing operation performed by the user isequal to or longer than a reference time period used in the presentaugmented reality process, the augmented reality processing unit 142determines that the transition condition 1 is established, and can thuscause a state (FIG. 4) of the augmented reality process to transitionfrom an initial state to the normal display state ST1.

As described above, according to the augmented reality process(transition from an initial state to the normal display state ST1 due toestablishment of the transition condition 1), the augmented realityprocessing unit 142 causes the image display section 20 to form thevirtual image VI (NI) including the virtual objects (VO1 to VO3 in FIG.6) in the first display aspect in relation to a real object (targetobject) on which at least a focusing operation is performed when thefocusing operation is continuously performed for a predeterminedreference time period. In the above-described way, since the virtualobject is displayed according to a user's intention such as a continuousfocusing operation, a user can maintain a state in which a real objectwhich actually exists in the real world is easily visually recognized aslong as the user does not continuously perform the focusing operation.As a result, it is possible to provide a head mounted display (the HMD100) in which display of a virtual object is unlikely to hinder visualrecognition of a real object or the background thereof.

In addition, according to the augmented reality process (monitoring ofestablishment of the transition condition 1), the augmented realityprocessing unit 142 can change a reference time period for switching adisplay aspect to the first display aspect (FIG. 6) in which an areaoccupied by the virtual objects VO1 to VO3 in the virtual image VI islarge (a degree of the visibility hindrance is high), for example,depending on various conditions as listed in the conditions 1-2 to 1-5after the augmented reality process is started.

In the augmented reality process (monitoring of establishment of thetransition condition 1), since a visual line of the user acquired by thevisual line acquisition unit (the visual line detection unit 62 and theaugmented reality processing unit 142 of the CPU 140) is used, the usercan perform a focusing operation by using the visual line without movingthe hand or the foot. For this reason, the user can easily perform afocusing operation even when performing work in which it is difficultfor the user to freely use his or her hand. In the augmented realityprocess (monitoring of establishment of the transition condition 1),since a motion of the user's hand acquired by the motion detection unit(the cameras 61 and the augmented reality processing unit 142 of the CPU140) is used, the user can easily perform a focusing operation by usinga motion of the hand which is familiar in a normal action.

A-2-5. Monitoring of Establishment of Transition Condition 3

Procedures of monitoring the establishment of the transition condition 3are nearly the same as those of the transition condition 1 illustratedin FIG. 12. Hereinafter, difference therebetween will be described.Among at least one of conditions 3-1 to 3-5 listed below is set in theaugmented reality processing unit 142 in advance as the transitioncondition 3. The augmented reality processing unit 142 determines thatthe transition condition 3 is established in a case where any one of theset conditions 3-1 to 3-5 is established.

(3-1) In a case where a real object which is a display target of avirtual object enters a visual field of the user, and the user performsa predetermined operation

(3-2) In a case where a focusing operation performed by the user iscontinuously performed for a time period which is equal to or longerthan a statistic of a reference time period in the past augmentedreality process

(3-3) In a case where a focusing operation performed by the user iscontinuously performed for a time period which is equal to or longerthan a time period which is obtained on the basis of an informationamount of a virtual object

(3-4) In a case where a focusing operation performed by the user iscontinuously performed for a time period which is equal to or longerthan a value which is set by the user

(3-5) In a case where a focusing operation performed by the user iscontinuously performed for a time period which is equal to or longerthan a time period which is obtained by taking into consideration anindividual preference of the user and an information amount of a virtualobject

(3-1) In a case where a real object which is a display target of avirtual object enters the visual field of the user, and the userperforms a predetermined operation

Processes in steps S310 and S312 are the same as those in FIG. 12. Ifthe target object is included in step S312 (step S312: YES), theaugmented reality processing unit 142 monitors whether or not apredetermined operation is performed by the user. As the predeterminedoperation, any operation may be employed as long as the operation isdifferent from the invalidation action described in FIG. 4, and, forexample, a specific “gesture” may be employed. A method of acquiring agesture is the same as a method of acquiring an invalidation action. Ifthe predetermined operation is performed, in step S314, the augmentedreality processing unit 142 determines that the transition condition 3is established.

As mentioned above, if the condition 3-1 is used, the augmented realityprocessing unit 142 can determine that the transition condition 3 isestablished in a case where a real object (target object) which is adisplay target of a virtual object enters a visual field of the user andthe user performs a predetermined operation, and can cause a state (FIG.4) of the augmented reality process to transition from the simpledisplay state ST2 to the normal display state ST1.

(3-2) In a case where a focusing operation performed by the user iscontinuously performed for a time period which is equal to or longerthan a statistic of a reference time period in the past augmentedreality process

Processes in steps S320 to S326 are the same as those in FIG. 12. Instep S328, the augmented reality processing unit 142 determines that thetransition condition 3 is established and then performs storing on thepast reference time 124 (in the same manner as in step S328). Next, theaugmented reality processing unit 142 specifies either of the followingreal objects d1 and d2. A specifying method is the same as in step S328.

(d1) A real object which is a target of the focusing operation

(d2) A real object which is a target of the focusing operation and iscorrelated with a virtual object in the second display aspect

As mentioned above, if the condition 3-2 is used, in the same manner asin the condition 1-2, the augmented reality processing unit 142 canautomatically change a reference time period used in the presentaugmented reality process on the basis of a tendency of a statistic of areference time period (past reference time period) used in the pastaugmented reality process. In a case where a continuous time period of afocusing operation of the user is equal to or longer than a referencetime period used in the present augmented reality process, the augmentedreality processing unit 142 determines that the transition condition 3is established and can thus cause a state (FIG. 4) of the augmentedreality process to transition from the simple display state ST2 to thenormal display state ST1.

(3-3) In a case where a focusing operation performed by the user iscontinuously performed for a time period which is equal to or longerthan a time period which is obtained on the basis of an informationamount of a virtual object

Processes in steps S330, S334 and S336 are the same as those in FIG. 12.In relation to steps S332 and S338, the description of the “real objectwhich is a target of the focusing operation” may be replaced with“either of the real objects d1 and d2 described in the condition 3-2”.

As mentioned above, if the condition 3-3 is used, in the same manner asin the condition 1-3, the augmented reality processing unit 142 canchange a reference time period used in the present augmented realityprocess on the basis of an information amount of a virtual object (VO1to VO3 in FIG. 6) in the first display aspect, which occupies a largearea in a virtual image VI (a degree of the visibility hindrance ishigh). In the above-described way, for example, the augmented realityprocessing unit 142 can make a reference time period in a case where aninformation amount of a virtual object in the first display aspect islarge, in other words, display of a virtual object in transition fromthe second display aspect (FIGS. 8, 9, 10 and 11) to the first displayaspect is likely to hinder visual recognition of a real object longerthan a reference time period in a case where an information amount issmall, and thus it is possible to improve a user's convenience. Inaddition, in a case where a continuous time period of a focusingoperation performed by the user is equal to or longer than a referencetime period used in the present augmented reality process, the augmentedreality processing unit 142 determines that the transition condition 3is established and can thus cause a state (FIG. 4) of the augmentedreality process to transition from the simple display state ST2 to thenormal display state ST1.

(3-4) In a case where a focusing operation performed by the user iscontinuously performed for a time period which is equal to or longerthan a value which is set by the user

Processes in steps S340 to S344 are the same as those in FIG. 12. Inrelation to step S346, the description of the “real object which is atarget of the focusing operation” may be replaced with “either of thereal objects d1 and d2 described in the condition 3-2”.

As mentioned above, if the condition 3-4 is used, in the same manner asin the condition 1-4, the augmented reality processing unit 142 canchange a reference time period used in the present augmented realityprocess on the basis of a set value which is preferred by the user andis stored in the reference time setting 125. In addition, in a casewhere a continuous time period of a focusing operation performed by theuser is equal to or longer than a reference time period used in thepresent augmented reality process, the augmented reality processing unit142 determines that the transition condition 3 is established, and canthus cause a state (FIG. 4) of the augmented reality process totransition from the simple display state ST2 to the normal display stateST1.

(3-5) In a case where a focusing operation performed by the user iscontinuously performed for a time period which is equal to or longerthan a time period which is obtained by taking into consideration anindividual preference of the user and an information amount of a virtualobject

Processes in steps S350 to S356 are the same as those in FIG. 12. Inrelation to step S358, the description of the “real object which is atarget of the focusing operation” may be replaced with “either of thereal objects d1 and d2 described in the condition 3-2”.

As mentioned above, if the condition 3-5 is used, in the same manner asin the condition 1-5, the augmented reality processing unit 142 canchange a reference time period used in the present augmented realityprocess on the basis of, for example, an information amount in which theuser can focus per unit time and an information amount of a virtualobject (VO1 to VO3 in FIG. 6) in the first display aspect. In theabove-described way, the augmented reality processing unit 142 can makea reference time period in a case where an information amount in whichthe user can focus per unit time is small, in other words, display of avirtual object in transition from the second display aspect (FIGS. 8, 9,10 and 11) to the first display aspect is likely to hinder visualrecognition of a real object longer than a reference time period in acase where an information amount is large. As a result, the augmentedreality processing unit 142 can change a reference time period accordingto an individual preference of the user, and thus it is possible toimprove a user's convenience. In addition, in a case where a continuoustime period of a focusing operation performed by the user is equal to orlonger than a reference time period used in the present augmentedreality process, the augmented reality processing unit 142 determinesthat the transition condition 3 is established, and can thus cause astate (FIG. 4) of the augmented reality process to transition from thesimple display state ST2 to the normal display state ST1.

As described above, according to the augmented reality process(transition from the simple display state ST2 to the normal displaystate ST1 due to establishment of the transition condition 3), after theimage display section 20 forms the virtual image VI (SI) including thevirtual objects (VO4 to VO6 in FIGS. 8, 9, 10 and 11) in the seconddisplay aspect, the augmented reality processing unit 142 can cause theimage display section 20 to form the virtual image VI (NI) including thevirtual objects (VO1 to VO3 in FIG. 6) in the first display aspect,having a larger occupied area by the virtual object (a high degree ofthe visibility hindrance) in the virtual image than an area of virtualobjects in the second display aspect when the focusing operation iscontinuously performed on a virtual object (d2) for a predeterminedreference time period in addition to a real object (d1). In theabove-described way, since a display aspect of the virtual objecttransitions from the second display aspect to the first display aspectaccording to a user's intention such as a continuous focusing operation,the user can maintain a state in which a real object which actuallyexists in the real world is easily visually recognized as long as theuser does not continuously perform the focusing operation. In otherwords, the user can control a degree of the visibility hindrance of avirtual object according to the user's intention. As a result, it ispossible to provide a head mounted display (the HMD 100) in whichdisplay of a virtual object is unlikely to hinder visual recognition ofa real object or the background thereof.

In addition, according to the augmented reality process (monitoring ofestablishment of the transition condition 3), the augmented realityprocessing unit 142 can change a reference time period for switching adisplay aspect from the second display aspect (FIGS. 8, 9, 10 and 11) inwhich an area occupied by the virtual objects VO4 to VO6 in the virtualimage VI is small (a degree of the visibility hindrance is low), to thefirst display aspect (FIG. 6) in which an area occupied by the virtualobjects VO1 to VO3 in the virtual image VI is large (a degree of thevisibility hindrance is high), for example, depending on variousconditions as listed in the conditions 3-2 to 3-5.

A-2-6. Monitoring of Establishment of Transition Condition 2

FIG. 13 is a flowchart illustrating procedures of monitoring theestablishment of the transition condition 2. Among at least one ofconditions 2-1 to 2-5 listed below is set in the augmented realityprocessing unit 142 in advance as the transition condition 2. Theaugmented reality processing unit 142 determines that the transitioncondition 2 is established in a case where any one of the set conditions2-1 to 2-5 is established.

(2-1) In a case where a real object which is a display target of avirtual object comes out of a visual field of the user

(2-2) In a case where time which is equal to or greater than a statisticof a retention time period in the past augmented reality processes haselapsed after a virtual object in the first display aspect is displayed

(2-3) In a case where time which is equal to or greater than a timeperiod obtained from an information amount of a virtual object haselapsed after the virtual object in the first display aspect isdisplayed

(2-4) In a case where time which is equal to or greater than a value setby the user has elapsed after a virtual object in the first displayaspect is displayed

(2-5) In a case where time which is equal to or greater than a timeperiod obtained by taking into consideration an individual preference ofthe user and an information amount of a virtual object has elapsed afterthe virtual object in the first display aspect is displayed

Then, with reference to FIG. 13, a description will be made of specificprocedures for determining establishment of the conditions 2-1 to 2-5 inthe augmented reality processing unit 142.

(2-1) CASE (out of visual field): In a case where a real object which isa display target of a virtual object comes out of a visual field of theuser

In step S410, the augmented reality processing unit 142 acquires anexternal scenery image by using the cameras 61. In step S412, theaugmented reality processing unit 142 performs image recognition on theacquired external scenery image so as to determine whether or not a realobject for a virtual object which is being displayed in the firstdisplay aspect on the external scenery image is included in the externalscenery image. The “real object which is a display target of a virtualobject” is a “target object” in the simple display process (FIG. 7).

If the target object is not included therein (step S412: NO), theaugmented reality processing unit 142 determines that the transitioncondition 2 is established in step S414. If the target object isincluded therein, the augmented reality processing unit 142 continues tomonitor establishment of the conditions 2-1 to 2-5.

As mentioned above, if the condition 2-1 is used, the augmented realityprocessing unit 142 can determine that the transition condition 2 isestablished in a case where a real object for a virtual object which isdisplayed in the first display aspect comes out of a visual field of theuser, and can cause a state (FIG. 4) of the augmented reality process totransition from the normal display state ST1 to the simple display stateST2. As a result, the simple display process (FIG. 7) is performed bythe simple display processing portion 146, and the HMD 100 allows theuser to visually recognize a virtual image VI (FIGS. 8, 9, 10 and 11)including a virtual object in the second display aspect.

(2-2) CASE (past retention time): In a case where time which is equal toor greater than a statistic of a retention time period in the pastaugmented reality process has elapsed after a virtual object in thefirst display aspect is displayed

In step S420, the augmented reality processing unit 142 acquires notonly history of a retention time period (hereinafter, also referred toas a “past retention time period”) used in the past augmented realityprocess, stored in the past retention time 122 but also historyassociated with other users. In step S420, the augmented realityprocessing unit 142 functions as a “retention time acquisition unit”,and the past retention time 122 functions as “retention timeinformation”.

In step S422, the augmented reality processing unit 142 obtains astatistic of the past retention time periods by using the acquiredhistory. The statistic may be obtained by using any statistical method,and may be, for example, an average value, a mode, or a median. Theaugmented reality processing unit 142 sets the obtained statistic as a“retention time period used in the present process”.

In step S424, the augmented reality processing unit 142 determineswhether or not a display time period of the normal display image NI(FIG. 6) of which measurement is started in step S114 of FIG. 5 is equalto or longer than the retention time period (the statistic in step S422)used in the present process. The display time period of the normaldisplay image NI (FIG. 6) is a display time period of a virtual objectin the first display aspect.

If the display time period is equal to or greater than the statistic(step S424: YES), in step S426, the augmented reality processing unit142 determines that the transition condition 2 is established. If thedisplay time period is less than the statistic, the augmented realityprocessing unit 142 continues to monitor establishment of the conditions2-1 to 2-5.

As mentioned above, if the condition 2-2 is used, the augmented realityprocessing unit 142 can automatically change a retention time periodused in the present augmented reality process on the basis of astatistic of a retention time period (past retention time period) usedin the past augmented reality process, that is, a tendency of the pastretention time period. In a case where a display time period of thenormal display image NI (a display time period of the virtual objectsVO1 to VO3 in the first display aspect) is equal to or longer than aretention time period used in the present augmented reality process, theaugmented reality processing unit 142 determines that the transitioncondition 2 is established and can thus cause a state (FIG. 4) of theaugmented reality process to transition from the normal display stateST1 to the simple display state ST2.

(2-3) CASE (information amount): In a case where time which is equal toor greater than a time period obtained from an information amount of avirtual object has elapsed after the virtual object in the first displayaspect is displayed

In step S430, the augmented reality processing unit 142 acquires aninformation amount of a virtual object which is being displayed in thefirst display aspect. A method of acquiring an information amount of avirtual object is the same as the methods c1 to c3 in step S332 of FIG.12.

In step S432, the augmented reality processing unit 142 obtains athreshold value of a retention time period used in the present processon the basis of the information amount of the virtual object acquired instep S430. The threshold value may be obtained by using any method, and,for example, a value obtained by multiplying an information amount by apredetermined coefficient (a coefficient which is different from thecoefficient in step S334 of FIG. 12) may be used as the threshold value,and the threshold value may be obtained by using a table (a table whichis different from the table in step S334 of FIG. 12) in whichinformation amount candidates are correlated with threshold valuecandidates. The augmented reality processing unit 142 sets the obtainedthreshold value as a “retention time period used in the presentprocess”.

In step S434, the augmented reality processing unit 142 determineswhether or not a display time period of the normal display image NI(FIG. 6) of which measurement is started in step S114 of FIG. 5 is equalto or longer than the retention time period (the threshold value in stepS432) used in the present process.

If the display time period is equal to or greater than the thresholdvalue (step S434: YES), in step S436, the augmented reality processingunit 142 determines that the transition condition 2 is established. Ifthe display time period is smaller than the threshold value, theaugmented reality processing unit 142 continues to monitor establishmentof the conditions 2-1 to 2-5.

As mentioned above, if the condition 2-3 is used, the augmented realityprocessing unit 142 can change a retention time period used in thepresent augmented reality process on the basis of an information amountof a virtual object (VO1 to VO3 in FIG. 6) in the first display aspect,which occupies a large area in a virtual image VI (a degree of thevisibility hindrance is high). In the above-described way, for example,the augmented reality processing unit 142 can make a retention timeperiod in a case where an information amount of a virtual object in thefirst display aspect is large, in other words, it is estimated that theuser requires much time to understand content of a virtual object longerthan a retention time period in a case where an information amount issmall, and thus it is possible to improve a user's convenience. Inaddition, in a case where a display time period of a virtual object inthe first display aspect is equal to or longer than a retention timeperiod used in the present augmented reality process, the augmentedreality processing unit 142 determines that the transition condition 2is established, and can thus cause a state (FIG. 4) of the augmentedreality process to transition from the normal display state ST1 to thesimple display state ST2.

(2-4) CASE (user's setting): In a case where time which is equal to orgreater than a value set by the user has elapsed after a virtual objectin the first display aspect is displayed

In step S440, the augmented reality processing unit 142 acquires a setvalue of a retention time period which is set by the user, and stored inthe retention time setting 123. The augmented reality processing unit142 functions as a “retention time acquisition unit” in step S440. Theaugmented reality processing unit 142 sets the acquired set value as a“retention time period used in the present process”.

In step S442, the augmented reality processing unit 142 determineswhether or not a display time period of the normal display image NI(FIG. 6) of which measurement is started in step S114 of FIG. 5 is equalto or longer than the retention time period (the set value in step S440)used in the present process.

If the display time period is equal to or greater than the set value(step S442: YES), in step S444, the augmented reality processing unit142 determines that the transition condition 2 is established. If thedisplay time period is smaller than the threshold value, the augmentedreality processing unit 142 continues to monitor establishment of theconditions 2-1 to 2-5.

As mentioned above, if the condition 2-4 is used, the augmented realityprocessing unit 142 can change a retention time period used in thepresent augmented reality process on the basis of a set value which ispreferred by the user and is stored in the retention time setting 123.In addition, in a case where a display time period of a virtual objectin the first display aspect is equal to or longer than a retention timeperiod used in the present augmented reality process, the augmentedreality processing unit 142 determines that the transition condition 2is established, and can thus cause a state (FIG. 4) of the augmentedreality process to transition from the normal display state ST1 to thesimple display state ST2.

(2-5) CASE (individual preference of the user): In a case where timewhich is equal to or greater than a time period obtained by taking intoconsideration an individual preference of the user and an informationamount of a virtual object has elapsed after the virtual object in thefirst display aspect is displayed

In step S450, the augmented reality processing unit 142 acquires historyassociated with the present user of the HMD 100 on the basis of historyof a retention time period used in the past augmented reality process,stored in the past retention time 122. The augmented reality processingunit 142 may search the past retention time 122 with an identifier ofthe user as a key. The augmented reality processing unit 142 functionsas a “retention time acquisition unit” in step S450, and the pastretention time 122 functions as “retention time information”.

In step S452, the augmented reality processing unit 142 obtains aninformation amount in which the user of the HMD 100 can performrecognition per unit time by dividing an “information amount” of theacquired history by the “retention time period”. Next, the augmentedreality processing unit 142 obtains an ideal retention time period bydividing an information amount of a virtual object in the first displayaspect by the obtained information amount (the information amount inwhich the user can perform recognition per unit time). The augmentedreality processing unit 142 sets the obtained ideal retention timeperiod as a “retention time period used in the present process”. Amethod of obtaining an information amount of a virtual object is thesame as the methods c1 to c3 in step S332 of FIG. 12.

In step S454, the augmented reality processing unit 142 determineswhether or not a display time period of the normal display image NI(FIG. 6) of which measurement is started in step S114 of FIG. 5 is equalto or longer than the retention time period (the ideal retention timeperiod in step S452) used in the present process.

If the display time period is equal to or greater than the idealretention time period (step S454: YES), in step S456, the augmentedreality processing unit 142 determines that the transition condition 2is established. If the display time period is smaller than the idealretention time period, the augmented reality processing unit 142continues to monitor establishment of the conditions 2-1 to 2-5.

As mentioned above, if the condition 2-5 is used, the augmented realityprocessing unit 142 can obtain an information amount in which the usercan perform recognition per unit time by using the retention timeinformation (the past retention time 122). For this reason, theaugmented reality processing unit 142 can change a retention time periodused in the present augmented reality process on the basis of, forexample, the obtained information amount (an information amount in whichthe user can perform recognition per unit time) and an informationamount of a virtual object (VO1 to VO3 in FIG. 6) in the first displayaspect. In the above-described way, the augmented reality processingunit 142 can make a retention time period in a case where an informationamount in which the user can perform recognition per unit time is small,in other words, it is estimated that the user requires much time tounderstand content of a virtual object in the first display aspectlonger than a retention time period in a case where an informationamount is large. As a result, the augmented reality processing unit 142can change a retention time period according to an individual preferenceof the user, and thus it is possible to improve a user's convenience. Inaddition, in a case where a display time period of a virtual object inthe first display aspect is equal to or longer than a retention timeperiod used in the present augmented reality process, the augmentedreality processing unit 142 determines that the transition condition 2is established, and can thus cause a state (FIG. 4) of the augmentedreality process to transition from the normal display state ST1 to thesimple display state ST2.

As described above, according to the augmented reality process(transition from the normal display state ST1 to the simple displaystate ST2 due to establishment of the transition condition 2), theaugmented reality processing unit 142 causes the image display section20 to form the virtual image VI (NI) including the virtual objects (VO1to VO3 in FIG. 6) in the first display aspect, and then causes the imagedisplay section 20 to form the virtual image VI (SI) including thevirtual objects (VO4 to VO6 in FIGS. 8, 9, 10 and 11) in the seconddisplay aspect, having a smaller occupied area (a low degree of thevisibility hindrance) in the virtual image than an area of the virtualobjects in the first display aspect after a predetermined retention timeperiod has elapsed. In the above-described way, since an area occupiedby the virtual object in the displayed virtual image is automaticallyreduced (a degree of the visibility hindrance is reduced) after theretention time period has elapsed, it becomes easier for the user tovisually recognize a real object which actually exists in the realworld. As a result, it is possible to provide a head mounted display(the HMD 100) in which display of a virtual object is unlikely to hindervisual recognition of a real object or the background thereof.

In addition, according to the augmented reality process (monitoring ofestablishment of the transition condition 2), the augmented realityprocessing unit 142 can change a retention time period for switching adisplay aspect from the first display aspect (FIG. 6) in which an areaoccupied by the virtual objects VO1 to VO3 in the virtual image VI islarge (a degree of the visibility hindrance is high) to the seconddisplay aspect (FIGS. 8, 9, 10 and 11) in which an area occupied by thevirtual objects VO4 to VO6 in the virtual image VI is small (a degree ofthe visibility hindrance is low), for example, depending on variousconditions as listed in the conditions 2-2 to 2-5.

B. Modification Examples

In the above-described embodiments, some of the constituent elementsrealized in hardware may be realized in software, and, conversely, someof the constituent elements realized in software may be realized inhardware. In addition, the following modifications may also occur.

Modification Example 1

In the above-described embodiment, a configuration of the HMD has beenexemplified. However, any configuration of the HMD may be defined withinthe scope without departing from the spirit of the invention, and, forexample, each configuration unit may be added, deleted, changed, or thelike.

In the above-described embodiment, the allocation of the constituentelements to the control section and the image display section are onlyan example, and may employ various aspects. For example, the followingaspects may be employed: (i) an aspect in which a processing functionsuch as a CPU and a memory is mounted in the control section, and only adisplay function is mounted in the image display section; (ii) an aspectin which a processing function such as a CPU and a memory is mounted inboth the control section and the image display section; (iii) an aspectin which the control section and the image display section areintegrally formed (for example, an aspect in which the image displaysection includes the control section and functions as a wearablecomputer); (iv) an aspect in which a smartphone or a portable gamemachine is used instead of the control section; and (v) an aspect inwhich the control section and the image display section are coupled toeach other via a wireless signal transmission path such as a wirelessLAN, infrared communication, or Bluetooth (registered trademark) so thatthe coupling unit (cords) is removed. In this case, the control sectionor the image display section may be supplied with power in a wirelessmanner.

For example, configurations of the control section and the image displaysection described in the embodiments may be arbitrarily changed.Specifically, in the above-described embodiment, the control section isprovided with the transmission unit, and the image display section isprovided with the reception unit. However, both of the transmission unitand the reception unit have a bidirectional communication function, andthus can function as a transmission and reception unit. For example,some of the operation interfaces (the various keys, the track pad, andthe like) included in the control section may be omitted. The controlsection may be provided with other operation interfaces such as anoperation stick. The control section may be configured to be coupled todevices such as a keyboard or a mouse, and may receive input from thekeyboard or the mouse. For example, a secondary battery is used as thepower supply, but the power supply is not limited to the secondarybattery and may use various batteries. For example, a primary battery, afuel cell, a solar cell, and a thermal cell may be used.

FIGS. 14A and 14B are diagrams illustrating exterior configurations ofHMDs in a modification example. In an example of FIG. 14A, an imagedisplay section 20 x includes a right optical image display unit 26 xinstead of the right optical image display unit 26 and a left opticalimage display unit 28 x instead of the left optical image display unit28. The right optical image display unit 26 x and the left optical imagedisplay unit 28 x are formed to be smaller than the optical members ofthe above-described embodiment, and are disposed on the oblique upperside of the right eye and the left eye of the user when the HMD ismounted. In an example of FIG. 14B, an image display section 20 yincludes a right optical image display unit 26 y instead of the rightoptical image display unit 26 and a left optical image display unit 28 yinstead of the left optical image display unit 28. The right opticalimage display unit 26 y and the left optical image display unit 28 y areformed to be smaller than the optical members of the above-describedembodiment, and are disposed on the oblique lower side of the right eyeand the left eye of the user when the HMD is mounted. As mentionedabove, the optical image display units have only to be disposed near theeyes of the user. Any size of the optical member forming the opticalimage display units may be used, and the HMD may be implemented in anaspect in which the optical image display units cover only part of theeyes of the user; in other words, the optical image display units do notcompletely cover the eyes of the user.

For example, a description has been made that the respective processingunits (the image processing unit, the display control unit, theaugmented reality processing unit, and the like) included in the controlsection are realized by the CPU developing a computer program stored inthe ROM or the hard disk on the RAM and executing the program. However,these function units may be configured using an application specificintegrated circuit (ASIC) which is designed for realizing each of thecorresponding functions. Each processing unit may be disposed not in thecontrol section but in the image display section.

For example, the HMD is a binocular transmission type HMD, but may be amonocular HMD. The HMD may be a non-transmissive HMD through whichexternal scenery is blocked from being transmitted in a state in whichthe user wears the HMD, and may be configured as a video see-throughtype in which a camera is mounted on the non-transmissive HMD. As animage display section, instead of the image display section which isworn such as spectacles, a typical flat display device (a liquid crystaldisplay, a plasma display panel, an organic EL display, or the like) maybe used. Also in this case, coupling between the control section and theimage display section may be performed via a wired signal transmissionpath, and may be performed via a wireless signal transmission path. Inthe above-described manner, the control section may used as a remotecontroller of a typical flat display device. In addition, as an imagedisplay section, instead of the image display section which is worn suchas spectacles, other types of image display sections such as an imagedisplay section which is worn such as a cap, may be employed. Theearphone may employ an ear-mounted type or a head band type, or may beomitted. For example, a head-up display (HUD) may be configured to bemounted in a vehicle such as an automobile or an airplane, and othertransportation. For example, the HMD may be configured to be built intoa body protection tool such as a helmet.

For example, in the above-described embodiment, the image lightgeneration unit is configured using the backlight, the backlight controlportion, the LCD, the LCD control portion. However, the above aspect isonly an example. The image light generation unit may include aconfiguration unit for realizing other types along with thisconfiguration unit or instead of this configuration unit. For example,the image light generation unit may include an organicelectroluminescence (EL) display and an organic EL controller. Forexample, the image light generation unit may use a digital micromirrordevice or the like instead of the LCD. For example, the invention isapplicable to a laser retinal projective head mounted display.

Modification Example 2

In the above-described embodiment, an example of the augmented realityprocess has been described. However, the procedures of the processdescribed in the embodiment are only an example, and variousmodifications may occur. For example, some steps may be omitted, andother steps may be added. In addition, an order of executed steps may bechanged.

For example, although the augmented reality process is started from thenormal display state ST1 (that is, transition from an initial state tothe normal display state ST1), the augmented reality process may bestarted from the simple display state ST2 (that is, transition from aninitial state to the simple display state ST2).

For example, the augmented reality processing unit may also monitor aninvalidation action in the same manner as in the transition conditions 2and 3 during monitoring of the transition condition 1, so as to stopstate transition. For example, the augmented reality processing unit mayomit monitoring of an invalidation action during monitoring of thetransition conditions 2 and 3.

For example, the augmented reality processing unit may monitor a“focusing operation” realized by a motion of the user's head acquired bythe nine-axis sensor during monitoring of the transition conditions 1and 3, instead of the visual line acquisition unit (which acquires amotion of a visual line) or the motion acquisition unit (which acquiresa motion of the hand) described in the embodiment, or along with thevisual line acquisition unit or the motion acquisition unit.

For example, in the normal display process (FIG. 5), the normal displayprocessing portion may also display a virtual object for a real object(hereinafter, referred to as “another real object”) which is not atarget of a focusing operation of the user among a plurality of realobjects included in an external scenery image when the transitioncondition 1 or the transition condition 3 is established. The normaldisplay processing portion may change a display aspect of a virtualobject added to another real object and a display aspect of a virtualobject added to a real object (that is, a target object) which is atarget of the focusing operation to each other. The display aspect is,for example, size, brightness, or color saturation.

For example, a forced transition action (second request) for forcing theaugmented reality process to transition may be used instead of theabove-described invalidation action (first request) or along with theinvalidation action. The forced transition action is an operationperformed by using at least one of the hand, the foot, a sound and thehead of the user, and a combination thereof, and may employ anyoperation as long as the operation is different from the invalidationaction. In a case where the forced transition action is detected whenthe augmented reality process is in the normal display state ST1, theaugmented reality processing unit causes the augmented reality processto transition to the simple display state ST2. In the above-describedmanner, the augmented reality processing unit can force a display aspectof the augmented reality process to transition from the first displayaspect to the second display aspect even before a retention time periodhas elapsed in response to the second request from the user, and thus itis possible to improve a user's convenience. On the other hand, in acase where the forced transition action is detected when the augmentedreality process is in the simple display state ST2, the augmentedreality processing unit causes the augmented reality process totransition to the normal display state ST1. In the above-describedmanner, the augmented reality processing unit can display a virtualobject in the first display aspect in response to the second requestfrom the user even before a reference time period has elapsed, and thusit is possible to improve a user's convenience.

Modification Example 3

If e1 to e8 described below are performed on the augmented realityprocess exemplified in the embodiment, it is possible to realize worksupport using the HMD.

(e1) Information related to work (including, for example, an instructionfor work content, information for assisting the instruction for workcontent, standard time required in work, and information for specifyinga motion of the user during work) is stored in the storage unit inadvance.

(e2) A motion of the user's body is acquired by using the nine-axissensor and the cameras (the nine-axis sensor and the cameras may be usedalone or together with each other; and other sensors may also be used).

(e3) After work support is started, a progress condition of the workperformed by the user is monitored by combining the information relatedto the work (the standard time required in the work, and the informationfor specifying a motion of the user during the work) stored in thestorage unit with the motion of the user's body acquired in themodification e2.

(e4) In a case where a monitoring result of the modification e3indicates either “work stoppage” in which a motion of the user's bodystops or “work delay” in which progress is delayed by a predeterminedtime, it is determined that the transition condition 1 (a transitioncondition from an initial state to the normal display state ST1) of theaugmented reality process is established, and a virtual object in thefirst display aspect is displayed. The displayed virtual object isinformation based on the information related to the work (theinstruction for work content and the information for assisting theinstruction for work content) stored in the storage unit.

(e5) In a case where a monitoring result of the modification e3indicates “during work” in which progress is performed as scheduled inthe normal display state ST1, it is determined that the transitioncondition 2 (a transition condition from the normal display state ST1 tothe simple display state ST2) of the augmented reality process isestablished, and a virtual object in the second display aspect isdisplayed. The virtual object in the second display aspect may not bedisplayed, and may be displayed by using an icon image, text, or thelike, as described above.

(e6) In a case where a monitoring result of the modification e3indicates the work stoppage or the work delay in the normal displaystate ST1, the normal display state ST1 is retained.

(e7) In a case where a monitoring result of the modification e3indicates the work stoppage or the work delay in the simple displaystate ST2, it is determined that the transition condition 3 (atransition condition from the simple display state ST2 to the normaldisplay state ST1) of the augmented reality process is established, anda virtual object in the first display aspect is displayed.

(e8) In a case where a monitoring result of the modification e3indicates during work in the simple display state ST2, the simpledisplay state ST2 is retained.

In the above-described manner, the HMD can cause the simple displaystate ST2 (including display and non-display of a virtual object in thesecond display aspect) to be retained in a case of a skilled worker whocan perform smooth work and can cause the normal display state ST1(display of a virtual object in the first display aspect) to be retainedin a case of an unskilled worker unaccustomed to the work. Even in acase of the skilled worker, when the work is stopped due to theoccurrence of unclarity in the middle of procedures, transition to thenormal display state ST1 may occur, and a virtual object in the firstdisplay aspect may be displayed. As a result, it is possible to providean HMD which can perform work support in which a worker's convenience isimproved. As a result, in a case where a virtual object may not berequired to be displayed, such as a case where a skilled worker is auser of the HMD, display of the virtual object may be omitted (orsimplified). For this reason, it is possible to reduce a possibilitythat the visibility of a real object may unnecessarily deteriorate andthus to reduce concern that the user may experience inconvenience.

As described above, in Modification Example 3, in a case where apredetermined operation (information for specifying a motion of the userduring work) is not started by the user (worker) within a predeterminedreference time period (the standard time required in the work), theaugmented reality processing unit causes the image display section toform a virtual image including a virtual object in the first displayaspect. In other words, in a case where the predetermined operation isstarted by the user within the reference time period, the augmentedreality processing unit does not display the virtual object in the firstdisplay aspect. For this reason, for example, in a case where the userperforms a predetermined operation (for example, certain work), it ispossible to reduce a possibility that a virtual object in the firstdisplay aspect may be displayed and may shield a user's field of vision.As a result, it is possible to provide a head mounted display (HMD) inwhich display of a virtual object is unlikely to hinder visualrecognition of a real object or the background thereof.

Also in Modification Example 3, in the same manner as in the embodiment,transition between the normal display state ST1 and the simple displaystate ST2 may be performed when a focusing operation is continuouslyperformed during a predetermined reference time period. In theabove-described way, in a case where a worker who performs a series ofwork items is at a loss since the worker has no idea of the nextoperation, it is possible to perform work support through statetransition from the simple display state ST2 to the normal display stateST1.

In the above description that “the image display section forms a virtualimage including a virtual object in the first display aspect in a casewhere a predetermined operation is not started by the user within apredetermined reference time period”, this matches the content of theembodiment if the “predetermined operation” is replaced with an“operation for stopping a continuous focusing operation”.

Modification Example 4

The invention is not limited to the above-described embodiment ormodification examples, and may be implemented using variousconfigurations within the scope without departing from the spiritthereof. For example, the embodiment corresponding to technical featuresof the respective aspects described in the Summary of Invention and thetechnical features in the modification examples may be exchanged orcombined as appropriate in order to solve some or all of theabove-described problems, or in order to achieve some or all of theabove-described effects. In addition, if the technical feature is notdescribed as an essential feature in the present specification, thetechnical feature may be deleted as appropriate.

The entire disclosure of Japanese Patent Application Nos. 2014-212725,filed Oct. 17, 2014 and 2014-212728, filed Oct. 17, 2014 are expresslyincorporated by reference herein.

What is claimed is:
 1. A head mounted display which allows a user tovisually recognize a virtual image, comprising: an image display unitthat enables the user to visually recognize the virtual image; and anaugmented reality processing unit that causes the image display unit toform the virtual image including a virtual object which is displayedadditionally to a real object actually existing in the real world,wherein the augmented reality processing unit causes the virtual imageincluding the virtual object in a first display aspect to be formed, andthen causes the virtual image including the virtual object in a seconddisplay aspect to be formed after a predetermined retention time periodhas elapsed, and wherein a degree of the visibility hindrance of thevirtual object in the second display aspect for the real object is lowerthan a degree of the visibility hindrance of the virtual object in thefirst display aspect for the real object.
 2. The head mounted displayaccording to claim 1, wherein the retention time period has a variablelength.
 3. The head mounted display according to claim 2, furthercomprising: a retention time acquisition unit that acquires theretention time period used in the augmented reality processing unit inthe past, wherein the augmented reality processing unit obtains astatistic of the acquired past retention time period, and changes theretention time period used in the present process on the basis of theobtained statistic.
 4. The head mounted display according to claim 2,wherein the augmented reality processing unit obtains an informationamount of the virtual object in the first display aspect, and changesthe retention time period used in the present process on the basis ofthe obtained information amount.
 5. The head mounted display accordingto claim 4, wherein the augmented reality processing unit changes amethod of obtaining the information amount depending on the kind ofvirtual object in the first display aspect.
 6. The head mounted displayaccording to claim 2, further comprising: a retention time acquisitionunit that acquires a user's setting performed on the retention timeperiod, wherein the augmented reality processing unit changes theretention time period used in the present process on the basis of theacquired user's setting.
 7. The head mounted display according to claim2, further comprising: a retention time acquisition unit that acquiresretention time information in which the retention time period used inthe past in the augmented reality processing unit, an information amountof the virtual object in the first display aspect at that time, andidentification information for identifying the user are correlated witheach other, wherein the augmented reality processing unit changes theretention time period used in the present process on the basis of theacquired retention time information and the information amount of thevirtual object in the first display aspect.
 8. The head mounted displayaccording to claim 1, wherein the virtual object in the second displayaspect includes at least one of text, a graphic, a pattern, a symbol,and a combination thereof, suggesting content of the virtual object inthe first display aspect.
 9. The head mounted display according to claim1, wherein the augmented reality processing unit stops transition fromthe first display aspect to the second display aspect in a case where afirst request is acquired from the user while waiting for the retentiontime period to elapse.
 10. The head mounted display according to claim1, wherein the augmented reality processing unit allows transition fromthe first display aspect to the second display aspect even before theretention time period has elapsed in a case where a second request isacquired from the user while waiting for the retention time period toelapse.
 11. The head mounted display according to claim 9, furthercomprising: a request acquisition unit that acquires a request realizedby at least one of the hand, the foot, a sound and the head of the user,and a combination thereof, as the first request or the second request.12. The head mounted display according to claim 1, wherein the augmentedreality processing unit changes transition from the first display aspectto the second display aspect in stages.
 13. A control method for a headmounted display, the method comprising: allowing a user to visuallyrecognize a virtual image; and forming the virtual image including avirtual object which is displayed additionally to a real object actuallyexisting in the real world, wherein, in the forming of the virtualimage, the virtual image including the virtual object in a first displayaspect is formed, and then the virtual image including the virtualobject in a second display aspect is formed after a predeterminedretention time period has elapsed, and wherein a degree of thevisibility hindrance of the virtual object in the second display aspectfor the real object is lower than a degree of the visibility hindranceof the virtual object in the first display aspect for the real object.14. A computer program causing a computer to realize: a display functionof allowing a user to visually recognize a virtual image; and a controlfunction of causing the virtual image including a virtual object whichis displayed additionally to a real object actually existing in the realworld to be formed in the display function, wherein the control functioncauses the virtual image including the virtual object in a first displayaspect to be formed, and then causes the virtual image including thevirtual object in a second display aspect to be formed after apredetermined retention time period has elapsed, and wherein a degree ofthe visibility hindrance of the virtual object in the second displayaspect for the real object is lower than a degree of the visibilityhindrance of the virtual object in the first display aspect for the realobject.
 15. A head mounted display which allows a user to visuallyrecognize a virtual image and external scenery, comprising: an imagedisplay unit that enables the user to visually recognize the virtualimage; and an augmented reality processing unit that causes the imagedisplay unit to form the virtual image including a virtual object whichis displayed additionally to a real object actually existing in the realworld, wherein, in response to a continuous focusing operation on thereal object during a predetermined reference time period, the augmentedreality processing unit causes the virtual image including the virtualobject in a first display aspect to be formed, the virtual object beingrelated to the real object on which at least the focusing operation isperformed.
 16. The head mounted display according to claim 15, wherein,in a case where the virtual image including the virtual object in asecond display aspect is formed prior to formation of the virtual imageincluding the virtual object in the first display aspect, in response toa continuous focusing operation on either in the virtual object in thesecond display aspect or the real object during the reference timeperiod, the augmented reality processing unit further causes the virtualimage including the virtual object in the first display aspect to beformed, the virtual object being related to the virtual object or thereal object on which at least the focusing operation is performed, andwherein a degree of the visibility hindrance of the virtual object inthe second display aspect for the real object is lower than a degree ofthe visibility hindrance of the virtual object in the first displayaspect for the real object.
 17. The head mounted display according toclaim 15, further comprising: a visual line acquisition unit thatacquires a motion of a visual line of the user as the focusingoperation.
 18. The head mounted display according to claim 15, furthercomprising: a motion acquisition unit that acquires a motion of the handof the user as the focusing operation.
 19. The head mounted displayaccording to claim 15, wherein the reference time period has a variablelength.
 20. The head mounted display according to claim 19, furthercomprising: a reference time acquisition unit that acquires thereference time period used in the augmented reality processing unit inthe past, wherein the augmented reality processing unit obtains astatistic of the acquired past reference time period, and changes thereference time period used in the present process on the basis of theobtained statistic.
 21. The head mounted display according to claim 19,wherein the augmented reality processing unit obtains an informationamount of the virtual object in the first display aspect, and changesthe reference time period used in the present process on the basis ofthe obtained information amount.
 22. The head mounted display accordingto claim 21, wherein the augmented reality processing unit changes amethod of obtaining the information amount depending on the kind ofvirtual object in the first display aspect.
 23. The head mounted displayaccording to claim 19, further comprising: a reference time acquisitionunit that acquires a user's setting performed on the reference timeperiod, wherein the augmented reality processing unit changes thereference time period used in the present process on the basis of theacquired user's setting.
 24. The head mounted display according to claim19, further comprising: a reference time acquisition unit that acquiresreference time information in which the reference time period used inthe past in the augmented reality processing unit, an information amountof the virtual object in the first display aspect at that time, andidentification information for identifying the user are correlated witheach other, wherein the augmented reality processing unit changes thereference time period used in the present process on the basis of theacquired reference time information and the information amount of thevirtual object in the first display aspect.
 25. The head mounted displayaccording to claim 16, wherein the virtual object in the second displayaspect includes at least one of text, a graphic, a pattern, a symbol,and a combination thereof, suggesting content of the virtual object inthe first display aspect.
 26. The head mounted display according toclaim 15, wherein the augmented reality processing unit stops transitionto the first display aspect in a case where a first request is acquiredfrom the user while waiting for the reference time period to elapse. 27.The head mounted display according to claim 15, wherein the augmentedreality processing unit causes the virtual image including the virtualobject in the first display aspect to be formed even before thereference time period has elapsed in a case where a second request isacquired from the user while waiting for the reference time period toelapse.
 28. The head mounted display according to claim 26, furthercomprising: a request acquisition unit that acquires a request realizedby at least one of the hand, the foot, a sound and the head of the user,and a combination thereof, as the first request or the second request.29. A control method for a head mounted display, the method comprising:allowing a user to visually recognize a virtual image; and forming thevirtual image including a virtual object which is displayed additionallyto a real object actually existing in the real world, wherein, in theforming of the virtual image, in response to a continuous focusingoperation on the real object during a predetermined reference timeperiod, the virtual image including the virtual object in a firstdisplay aspect is formed, the virtual object being related to the realobject on which at least the focusing operation is performed.
 30. Acomputer program causing a computer to realize: a display function ofallowing a user to visually recognize a virtual image; and a controlfunction of causing the virtual image including a virtual object whichis displayed additionally to a real object actually existing in the realworld to be formed in the display function, wherein, in response to acontinuous focusing operation on the real object during a predeterminedreference time period, the control function causes the virtual imageincluding the virtual object in a first display aspect to be formed, thevirtual object being related to the real object on which at least thefocusing operation is performed.
 31. A head mounted display which allowsa user to visually recognize a virtual image and external scenery,comprising: an image display unit that enables the user to visuallyrecognize the virtual image; and an augmented reality processing unitthat causes the image display unit to form the virtual image including avirtual object which is displayed additionally to a real object actuallyexisting in the real world, wherein, in a case where a predeterminedoperation is not performed by the user within a predetermined referencetime period, the augmented reality processing unit causes the virtualimage including the virtual object in a first display aspect to beformed, the virtual object being related to at least the real object.